[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250113 21:10:01.524910 28045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.99.126:45585
I20250113 21:10:01.525200 28045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 21:10:01.525339 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 21:10:01.529671 28045 server_base.cc:1034] running on GCE node
W20250113 21:10:01.530119 28053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.530547 28051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.531188 28050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:01.531409 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:01.531466 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:01.531497 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802601531496 us; error 0 us; skew 500 ppm
I20250113 21:10:01.532521 28045 webserver.cc:458] Webserver started at http://127.27.99.126:40725/ using document root <none> and password file <none>
I20250113 21:10:01.532657 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:01.532709 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:01.532785 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:01.533684 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/master-0-root/instance:
uuid: "ef42ff0cfe3149a8a27e0757e790232d"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.534803 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:01.535327 28058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.535466 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:01.535528 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/master-0-root
uuid: "ef42ff0cfe3149a8a27e0757e790232d"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.535584 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:01.551144 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:01.551386 28045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 21:10:01.551501 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:01.551960 28064 maintenance_manager.cc:345] Maintenance manager is disabled (check --enable_maintenance_manager).
I20250113 21:10:01.555626 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.126:45585
I20250113 21:10:01.555667 28110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.126:45585 every 8 connection(s)
I20250113 21:10:01.556207 28111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:01.558774 28111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d: Bootstrap starting.
I20250113 21:10:01.559549 28111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:01.559784 28111 log.cc:826] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d: Log is configured to *not* fsync() on all Append() calls
I20250113 21:10:01.560281 28111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d: No bootstrap required, opened a new log
I20250113 21:10:01.561535 28111 raft_consensus.cc:357] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef42ff0cfe3149a8a27e0757e790232d" member_type: VOTER }
I20250113 21:10:01.561619 28111 raft_consensus.cc:383] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:01.561638 28111 raft_consensus.cc:738] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef42ff0cfe3149a8a27e0757e790232d, State: Initialized, Role: FOLLOWER
I20250113 21:10:01.561745 28111 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef42ff0cfe3149a8a27e0757e790232d" member_type: VOTER }
I20250113 21:10:01.561794 28111 raft_consensus.cc:397] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 21:10:01.561813 28111 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 21:10:01.561854 28111 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:01.562319 28111 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef42ff0cfe3149a8a27e0757e790232d" member_type: VOTER }
I20250113 21:10:01.562395 28111 leader_election.cc:304] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef42ff0cfe3149a8a27e0757e790232d; no voters:
I20250113 21:10:01.562529 28111 leader_election.cc:290] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 21:10:01.562584 28114 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:01.562736 28114 raft_consensus.cc:695] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 1 LEADER]: Becoming Leader. State: Replica: ef42ff0cfe3149a8a27e0757e790232d, State: Running, Role: LEADER
I20250113 21:10:01.562722 28111 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [sys.catalog]: configured and running, proceeding with master startup.
I20250113 21:10:01.562846 28114 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef42ff0cfe3149a8a27e0757e790232d" member_type: VOTER }
I20250113 21:10:01.563165 28115 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef42ff0cfe3149a8a27e0757e790232d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef42ff0cfe3149a8a27e0757e790232d" member_type: VOTER } }
I20250113 21:10:01.563184 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef42ff0cfe3149a8a27e0757e790232d. Latest consensus state: current_term: 1 leader_uuid: "ef42ff0cfe3149a8a27e0757e790232d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef42ff0cfe3149a8a27e0757e790232d" member_type: VOTER } }
I20250113 21:10:01.563333 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:01.563521 28122 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 21:10:01.563527 28115 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:01.563946 28122 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 21:10:01.564145 28045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 21:10:01.565165 28122 catalog_manager.cc:1348] Generated new cluster ID: ef5b2c89ced249a2ae7353bfbbe9bbe9
I20250113 21:10:01.565212 28122 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 21:10:01.577451 28122 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 21:10:01.577980 28122 catalog_manager.cc:1505] Loading token signing keys...
I20250113 21:10:01.582733 28122 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d: Generated new TSK 0
I20250113 21:10:01.582862 28122 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 21:10:01.595896 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:01.598035 28137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.598759 28135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.599721 28134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:01.599772 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:01.599902 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:01.599931 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:01.599961 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802601599961 us; error 0 us; skew 500 ppm
I20250113 21:10:01.605582 28045 webserver.cc:458] Webserver started at http://127.27.99.65:44171/ using document root <none> and password file <none>
I20250113 21:10:01.605664 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:01.605715 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:01.605762 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:01.606055 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-0-root/instance:
uuid: "571116d897274e4f98bcd671bf0bfe3b"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.606889 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:01.607350 28142 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.607513 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.607561 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-0-root
uuid: "571116d897274e4f98bcd671bf0bfe3b"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.607611 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:01.619163 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:01.619369 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:01.619550 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:01.619812 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:01.619840 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.619863 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:01.619890 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.624296 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.65:43365
I20250113 21:10:01.624346 28205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.65:43365 every 8 connection(s)
I20250113 21:10:01.624650 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:01.626114 28212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.626816 28209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.627413 28210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:01.628098 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:01.628239 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:01.628275 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:01.628294 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802601628294 us; error 0 us; skew 500 ppm
I20250113 21:10:01.629149 28045 webserver.cc:458] Webserver started at http://127.27.99.66:43913/ using document root <none> and password file <none>
I20250113 21:10:01.629269 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:01.629320 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:01.629357 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:01.629637 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-1-root/instance:
uuid: "76a522394a8041a18ebb4e478f89c29f"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.630278 28206 heartbeater.cc:346] Connected to a master server at 127.27.99.126:45585
I20250113 21:10:01.630362 28206 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:01.630511 28206 heartbeater.cc:510] Master 127.27.99.126:45585 requested a full tablet report, sending...
I20250113 21:10:01.630723 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:01.631057 28075 ts_manager.cc:194] Registered new tserver with Master: 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:01.631959 28075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39836
I20250113 21:10:01.632184 28219 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.632325 28045 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:01.632370 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-1-root
uuid: "76a522394a8041a18ebb4e478f89c29f"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.632408 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:01.647696 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:01.647964 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:01.648138 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:01.648339 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:01.648379 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.648417 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:01.648442 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.652498 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.66:38759
I20250113 21:10:01.652526 28282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.66:38759 every 8 connection(s)
I20250113 21:10:01.652933 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:01.654461 28287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.655186 28286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:01.655798 28289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:01.655959 28283 heartbeater.cc:346] Connected to a master server at 127.27.99.126:45585
I20250113 21:10:01.656021 28283 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:01.656113 28283 heartbeater.cc:510] Master 127.27.99.126:45585 requested a full tablet report, sending...
I20250113 21:10:01.656365 28075 ts_manager.cc:194] Registered new tserver with Master: 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:01.656401 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:01.656500 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:01.656525 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:01.656536 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802601656537 us; error 0 us; skew 500 ppm
I20250113 21:10:01.656808 28075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39844
I20250113 21:10:01.657182 28045 webserver.cc:458] Webserver started at http://127.27.99.67:42063/ using document root <none> and password file <none>
I20250113 21:10:01.657279 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:01.657330 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:01.657382 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:01.657670 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-2-root/instance:
uuid: "59ff47de1d4340e8b03823c58cb98738"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.658710 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:01.659281 28294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.659425 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:01.659469 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-2-root
uuid: "59ff47de1d4340e8b03823c58cb98738"
format_stamp: "Formatted at 2025-01-13 21:10:01 on dist-test-slave-4hc8"
I20250113 21:10:01.659516 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:01.666854 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:01.667055 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:01.667207 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:01.667368 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:01.667393 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.667424 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:01.667451 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.670852 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.67:45391
I20250113 21:10:01.670917 28358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.67:45391 every 8 connection(s)
I20250113 21:10:01.674389 28359 heartbeater.cc:346] Connected to a master server at 127.27.99.126:45585
I20250113 21:10:01.674450 28359 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:01.674556 28359 heartbeater.cc:510] Master 127.27.99.126:45585 requested a full tablet report, sending...
I20250113 21:10:01.674783 28075 ts_manager.cc:194] Registered new tserver with Master: 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391)
I20250113 21:10:01.675114 28075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39854
I20250113 21:10:01.675498 28045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004436313s
I20250113 21:10:01.684806 28045 test_util.cc:274] Using random seed: -568438973
I20250113 21:10:01.691939 28075 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39882:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 21:10:01.692202 28075 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 21:10:01.697980 28322 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecfe918cd2f49cfa3f0e1bf8d643e6a (DEFAULT_TABLE table=test-workload [id=e17c9601455f436299823501e92e6d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:01.698154 28322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecfe918cd2f49cfa3f0e1bf8d643e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:01.698418 28170 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecfe918cd2f49cfa3f0e1bf8d643e6a (DEFAULT_TABLE table=test-workload [id=e17c9601455f436299823501e92e6d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:01.698532 28170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecfe918cd2f49cfa3f0e1bf8d643e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:01.699671 28380 tablet_bootstrap.cc:492] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Bootstrap starting.
I20250113 21:10:01.699736 28247 tablet_service.cc:1467] Processing CreateTablet for tablet 3ecfe918cd2f49cfa3f0e1bf8d643e6a (DEFAULT_TABLE table=test-workload [id=e17c9601455f436299823501e92e6d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:01.699816 28247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecfe918cd2f49cfa3f0e1bf8d643e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:01.700148 28380 tablet_bootstrap.cc:654] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:01.700227 28381 tablet_bootstrap.cc:492] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Bootstrap starting.
I20250113 21:10:01.700690 28381 tablet_bootstrap.cc:654] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:01.701114 28384 tablet_bootstrap.cc:492] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Bootstrap starting.
I20250113 21:10:01.701294 28381 tablet_bootstrap.cc:492] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: No bootstrap required, opened a new log
I20250113 21:10:01.701380 28381 ts_tablet_manager.cc:1397] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:01.701529 28384 tablet_bootstrap.cc:654] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:01.701606 28380 tablet_bootstrap.cc:492] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: No bootstrap required, opened a new log
I20250113 21:10:01.701642 28380 ts_tablet_manager.cc:1397] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 21:10:01.701570 28381 raft_consensus.cc:357] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.701684 28381 raft_consensus.cc:383] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:01.701714 28381 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Initialized, Role: FOLLOWER
I20250113 21:10:01.701774 28380 raft_consensus.cc:357] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.701839 28380 raft_consensus.cc:383] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:01.701776 28381 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.701859 28380 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Initialized, Role: FOLLOWER
I20250113 21:10:01.701936 28380 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.702041 28359 heartbeater.cc:502] Master 127.27.99.126:45585 was elected leader, sending a full tablet report...
I20250113 21:10:01.702075 28380 ts_tablet_manager.cc:1428] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:01.702235 28206 heartbeater.cc:502] Master 127.27.99.126:45585 was elected leader, sending a full tablet report...
I20250113 21:10:01.702008 28381 ts_tablet_manager.cc:1428] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 21:10:01.702643 28384 tablet_bootstrap.cc:492] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: No bootstrap required, opened a new log
I20250113 21:10:01.702684 28384 ts_tablet_manager.cc:1397] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20250113 21:10:01.702845 28384 raft_consensus.cc:357] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.702908 28384 raft_consensus.cc:383] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:01.702944 28384 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Initialized, Role: FOLLOWER
I20250113 21:10:01.702991 28384 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.703130 28384 ts_tablet_manager.cc:1428] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:01.703186 28283 heartbeater.cc:502] Master 127.27.99.126:45585 was elected leader, sending a full tablet report...
I20250113 21:10:01.703557 28386 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 21:10:01.703608 28386 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.703805 28386 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:01.706806 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f" is_pre_election: true
I20250113 21:10:01.706892 28257 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 0.
I20250113 21:10:01.707065 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:01.707157 28386 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 21:10:01.707190 28386 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:01.707209 28386 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:01.707851 28386 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.707984 28386 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 1 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:01.708209 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:01.708274 28257 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:01.708874 28257 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 1.
I20250113 21:10:01.708961 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:01.709013 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:01.709040 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:01.709681 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 1.
I20250113 21:10:01.708971 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b" is_pre_election: true
I20250113 21:10:01.709877 28180 raft_consensus.cc:2371] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 1. Re-sending same reply.
I20250113 21:10:01.710067 28386 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:01.710189 28386 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 1 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:01.710239 28386 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:01.710765 28074 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 1 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:01.722529 28257 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 1 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250113 21:10:01.722530 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 1 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 21:10:01.722798 28398 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 21:10:01.722980 28386 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 21:10:01.739090 28045 test_util.cc:274] Using random seed: -568384688
I20250113 21:10:01.744387 28045 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250113 21:10:02.076810 28387 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:02.076930 28387 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:02.077598 28387 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.077812 28387 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 2 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:02.080296 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:02.080361 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:02.080605 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:02.080711 28387 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 21:10:02.087167 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 2 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 1. Current term is 2. Ops: []
I20250113 21:10:02.087419 28425 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 1.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
W20250113 21:10:02.087546 28425 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250113 21:10:02.087563 28425 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250113 21:10:02.102721 28425 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 21:10:02.102804 28425 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:02.102885 28425 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.103032 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:02.135835 28388 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:02.135927 28388 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:02.136574 28388 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.136757 28388 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 2 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:02.139289 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:02.139284 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:02.139443 28180 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 2: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:02.139955 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 2.
I20250113 21:10:02.140120 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:02.140224 28388 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 2 FOLLOWER]: Leader election won for term 2
I20250113 21:10:02.140323 28388 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 2 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:02.140381 28388 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.140810 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 1 to 2, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 2 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:02.221390 28387 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:02.221490 28387 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 2 FOLLOWER]: Advancing to term 3
I20250113 21:10:02.222177 28387 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.222344 28387 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 3 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:02.222558 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:02.222558 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:02.222656 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 2 FOLLOWER]: Advancing to term 3
I20250113 21:10:02.223222 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 3.
I20250113 21:10:02.223381 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:02.223482 28387 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 3 FOLLOWER]: Leader election won for term 3
I20250113 21:10:02.223582 28387 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 3 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:02.223623 28387 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.224025 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 2 to 3, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 3 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:02.227080 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 3 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 2. Current term is 3. Ops: []
I20250113 21:10:02.227294 28388 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250113 21:10:02.228461 28388 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 2 LEADER]: Stepping down as leader of term 2
I20250113 21:10:02.228493 28388 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:02.228554 28388 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.42, Last appended by leader: 42, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.228613 28388 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 2 FOLLOWER]: Advancing to term 3
I20250113 21:10:02.320703 28257 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 3 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 3 index: 42. (term mismatch)
I20250113 21:10:02.320791 28257 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 41
I20250113 21:10:02.320818 28257 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 2.42
I20250113 21:10:02.320842 28257 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:02.321070 28387 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250113 21:10:02.333683 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 3 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20250113 21:10:02.333961 28387 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250113 21:10:02.434868 28425 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:02.434960 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 3 FOLLOWER]: Advancing to term 4
I20250113 21:10:02.435606 28425 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.435768 28425 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 4 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:02.435953 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:02.435927 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:02.436177 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:02.436262 28425 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 21:10:02.453559 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 4 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 3. Current term is 4. Ops: []
I20250113 21:10:02.453830 28429 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 3.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250113 21:10:02.465440 28387 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 3 LEADER]: Stepping down as leader of term 3
I20250113 21:10:02.465505 28387 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:02.465577 28387 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 3.42, Last appended by leader: 42, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.465682 28387 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 3 FOLLOWER]: Advancing to term 4
I20250113 21:10:02.519948 28388 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:02.520046 28388 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 3 FOLLOWER]: Advancing to term 4
I20250113 21:10:02.520684 28388 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.520844 28388 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 4 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:02.521071 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:02.521070 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:02.521202 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 4: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:02.521757 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 4.
I20250113 21:10:02.522085 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:02.522243 28388 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 4 FOLLOWER]: Leader election won for term 4
I20250113 21:10:02.522312 28388 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 4 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:02.522369 28388 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 3.42, Last appended by leader: 42, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.522822 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 3 to 4, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 4 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:02.571870 28425 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:02.571956 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 4 FOLLOWER]: Advancing to term 5
I20250113 21:10:02.572562 28425 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.572721 28425 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 5 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:02.572888 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 5 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:02.572888 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 5 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:02.572999 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 4 FOLLOWER]: Advancing to term 5
I20250113 21:10:02.573534 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 5.
I20250113 21:10:02.573693 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:02.573807 28425 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 21:10:02.573907 28425 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 5 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:02.573971 28425 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 3.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.574407 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 4 to 5, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 5 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:02.623263 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 5 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 4. Current term is 5. Ops: []
I20250113 21:10:02.623502 28388 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250113 21:10:02.623593 28388 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 5.43 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20250113 21:10:02.633613 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 5 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 4. Current term is 5. Ops: []
I20250113 21:10:02.633853 28388 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250113 21:10:02.669786 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 5 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 3 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20250113 21:10:02.670081 28425 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250113 21:10:02.674960 28257 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 4 LEADER]: Stepping down as leader of term 4
I20250113 21:10:02.675019 28257 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:02.675065 28257 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.43, Last appended by leader: 43, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.675148 28257 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 4 FOLLOWER]: Advancing to term 5
I20250113 21:10:02.675736 28257 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 5 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 4 index: 43. Preceding OpId from leader: term: 5 index: 43. (term mismatch)
I20250113 21:10:02.675773 28257 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 42
I20250113 21:10:02.675786 28257 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 4.43
I20250113 21:10:02.675799 28257 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:02.675992 28425 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250113 21:10:02.771322 28387 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:02.771418 28387 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 5 FOLLOWER]: Advancing to term 6
I20250113 21:10:02.772096 28387 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.772253 28387 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 6 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:02.772396 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:02.772440 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:02.772642 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:02.772744 28387 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 21:10:02.785034 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 6 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 5. Current term is 6. Ops: []
I20250113 21:10:02.785262 28398 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 5.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250113 21:10:02.892505 28425 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 21:10:02.892616 28425 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:02.892695 28425 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 5.43, Last appended by leader: 43, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.892831 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 5 FOLLOWER]: Advancing to term 6
I20250113 21:10:02.898207 28387 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:02.898295 28387 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 6 FOLLOWER]: Advancing to term 7
I20250113 21:10:02.898909 28387 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.899092 28387 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 7 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:02.899303 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 7 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:02.899327 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 7 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:02.899398 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 6 FOLLOWER]: Advancing to term 7
I20250113 21:10:02.899957 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 7.
I20250113 21:10:02.900174 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:02.900275 28387 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 7 FOLLOWER]: Leader election won for term 7
I20250113 21:10:02.900331 28387 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 7 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:02.900377 28387 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 5.43, Last appended by leader: 43, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.900851 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 5 to 7, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 7 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:02.965754 28428 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:02.965842 28428 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 5 FOLLOWER]: Advancing to term 6
I20250113 21:10:02.966516 28428 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:02.966652 28428 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 6 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:02.966848 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:02.966853 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:02.966949 28332 raft_consensus.cc:2363] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f for earlier term 6. Current term is 7.
I20250113 21:10:02.967103 28222 leader_election.cc:400] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 6 election: Vote denied by peer 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391) with higher term. Message: Invalid argument: T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f for earlier term 6. Current term is 7.
I20250113 21:10:02.967144 28222 leader_election.cc:403] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250113 21:10:02.967209 28428 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 6 FOLLOWER]: Advancing to term 7
I20250113 21:10:02.967823 28428 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391) with higher term. Message: Invalid argument: T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f for earlier term 6. Current term is 7.
I20250113 21:10:03.015377 28257 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 7 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 7 index: 44. (index mismatch)
I20250113 21:10:03.015640 28387 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250113 21:10:03.018657 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 7 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 7 index: 44. (index mismatch)
I20250113 21:10:03.018849 28433 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250113 21:10:03.117569 28428 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:03.117678 28428 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 7 FOLLOWER]: Advancing to term 8
I20250113 21:10:03.118345 28428 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.118496 28428 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 8 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:03.118718 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 8 candidate_status { last_received { term: 7 index: 44 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:03.118719 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 8 candidate_status { last_received { term: 7 index: 44 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:03.118973 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:03.119086 28428 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 21:10:03.124202 28257 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 8 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 7. Current term is 8. Ops: []
I20250113 21:10:03.124423 28430 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 7.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250113 21:10:03.199240 28425 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:03.199335 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 7 FOLLOWER]: Advancing to term 8
I20250113 21:10:03.200038 28425 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.200217 28425 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 8 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:03.200431 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 8 candidate_status { last_received { term: 7 index: 44 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:03.200438 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 8 candidate_status { last_received { term: 7 index: 44 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:03.200546 28257 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 8: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:03.200727 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:03.200809 28425 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 21:10:03.208142 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 8 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 7. Current term is 8. Ops: []
I20250113 21:10:03.208349 28387 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 7.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250113 21:10:03.232081 28433 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 7 LEADER]: Stepping down as leader of term 7
I20250113 21:10:03.232165 28433 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:03.232259 28433 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 7.44, Last appended by leader: 44, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.232375 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 7 FOLLOWER]: Advancing to term 8
I20250113 21:10:03.233489 28428 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:03.233559 28428 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 8 FOLLOWER]: Advancing to term 9
I20250113 21:10:03.234192 28428 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.234321 28428 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 9 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:03.234539 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 9 candidate_status { last_received { term: 7 index: 44 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:03.234539 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 9 candidate_status { last_received { term: 7 index: 44 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:03.234629 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 8 FOLLOWER]: Advancing to term 9
I20250113 21:10:03.234638 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 8 FOLLOWER]: Advancing to term 9
I20250113 21:10:03.235245 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 9.
I20250113 21:10:03.235245 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 9.
I20250113 21:10:03.235447 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:03.235574 28428 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 9 FOLLOWER]: Leader election won for term 9
I20250113 21:10:03.235699 28428 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 9 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:03.235759 28428 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 7.44, Last appended by leader: 44, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.236223 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 7 to 9, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 9 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:03.312556 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 9 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 7 index: 44. Preceding OpId from leader: term: 9 index: 45. (index mismatch)
I20250113 21:10:03.312815 28428 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250113 21:10:03.326411 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 9 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 7 index: 44. Preceding OpId from leader: term: 9 index: 45. (index mismatch)
I20250113 21:10:03.326670 28428 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250113 21:10:03.578713 28425 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:03.578811 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 9 FOLLOWER]: Advancing to term 10
I20250113 21:10:03.579458 28425 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.579622 28425 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 10 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:03.579838 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 10 candidate_status { last_received { term: 9 index: 45 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:03.579843 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 10 candidate_status { last_received { term: 9 index: 45 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:03.580106 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:03.580201 28425 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250113 21:10:03.586221 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 10 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 9. Current term is 10. Ops: []
I20250113 21:10:03.586408 28435 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 9.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250113 21:10:03.633595 28435 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 9 LEADER]: Stepping down as leader of term 9
I20250113 21:10:03.633678 28435 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:03.633750 28435 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 9.45, Last appended by leader: 45, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.633895 28435 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 9 FOLLOWER]: Advancing to term 10
I20250113 21:10:03.688577 28425 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:03.688673 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 10 FOLLOWER]: Advancing to term 11
I20250113 21:10:03.689306 28425 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.689478 28425 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 11 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:03.689682 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 11 candidate_status { last_received { term: 9 index: 45 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:03.689687 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 11 candidate_status { last_received { term: 9 index: 45 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:03.689788 28257 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 10 FOLLOWER]: Advancing to term 11
I20250113 21:10:03.690300 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:03.690342 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 9 FOLLOWER]: Advancing to term 10
I20250113 21:10:03.690452 28257 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 11.
I20250113 21:10:03.690611 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:03.690694 28425 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 11 FOLLOWER]: Leader election won for term 11
I20250113 21:10:03.690799 28425 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 11 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:03.690853 28425 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 9.45, Last appended by leader: 45, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.690974 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.691108 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 10 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:03.691227 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 10 candidate_status { last_received { term: 9 index: 45 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:03.691298 28257 raft_consensus.cc:2363] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b for earlier term 10. Current term is 11.
I20250113 21:10:03.691283 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 10 candidate_status { last_received { term: 9 index: 45 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:03.691287 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 9 to 11, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 11 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:03.691496 28143 leader_election.cc:400] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 10 election: Vote denied by peer 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759) with higher term. Message: Invalid argument: T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b for earlier term 10. Current term is 11.
I20250113 21:10:03.691524 28143 leader_election.cc:403] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20250113 21:10:03.691583 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 10 FOLLOWER]: Advancing to term 11
I20250113 21:10:03.692030 28433 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759) with higher term. Message: Invalid argument: T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b for earlier term 10. Current term is 11.
I20250113 21:10:03.699518 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.700717 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.702838 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.706987 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.715148 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.731303 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.763492 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.786868 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 11 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 9 index: 45. Preceding OpId from leader: term: 11 index: 46. (index mismatch)
I20250113 21:10:03.787143 28425 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250113 21:10:03.788167 28257 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 11 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 9 index: 45. Preceding OpId from leader: term: 11 index: 46. (index mismatch)
I20250113 21:10:03.788385 28436 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250113 21:10:03.827728 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:03.888636 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:03.888731 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 11 FOLLOWER]: Advancing to term 12
I20250113 21:10:03.889370 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.889432 28435 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:03.889487 28435 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 11 FOLLOWER]: Advancing to term 12
I20250113 21:10:03.889535 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 12 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:03.889748 28257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 12 candidate_status { last_received { term: 11 index: 46 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:03.889767 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 12 candidate_status { last_received { term: 11 index: 46 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:03.890482 28435 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.890589 28257 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 12: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:03.890735 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:03.890744 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 12 candidate_status { last_received { term: 11 index: 46 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:03.890748 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 12 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 11. Current term is 12. Ops: []
I20250113 21:10:03.890857 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 12 candidate_status { last_received { term: 11 index: 46 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:03.890908 28180 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 12: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:03.890988 28433 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250113 21:10:03.891016 28425 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 11.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250113 21:10:03.891050 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:03.891062 28435 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 12 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:03.891114 28425 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 11 LEADER]: Stepping down as leader of term 11
I20250113 21:10:03.891144 28434 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250113 21:10:03.891167 28425 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:03.891219 28425 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 11.46, Last appended by leader: 46, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:03.891286 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 11 FOLLOWER]: Advancing to term 12
I20250113 21:10:03.956053 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:04.011606 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:04.011698 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 12 FOLLOWER]: Advancing to term 13
I20250113 21:10:04.012321 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.012481 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 13 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:04.012709 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 13 candidate_status { last_received { term: 11 index: 46 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:04.012706 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 13 candidate_status { last_received { term: 11 index: 46 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:04.012814 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 12 FOLLOWER]: Advancing to term 13
I20250113 21:10:04.012820 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 12 FOLLOWER]: Advancing to term 13
I20250113 21:10:04.013375 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 13.
I20250113 21:10:04.013459 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 13.
I20250113 21:10:04.013540 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters:
I20250113 21:10:04.013617 28433 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 13 FOLLOWER]: Leader election won for term 13
I20250113 21:10:04.013720 28433 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 13 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:04.013779 28433 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 11.46, Last appended by leader: 46, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.014240 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 11 to 13, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 13 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:04.133603 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 13 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 11 index: 46. Preceding OpId from leader: term: 13 index: 47. (index mismatch)
I20250113 21:10:04.133890 28433 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250113 21:10:04.134536 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 13 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 11 index: 46. Preceding OpId from leader: term: 13 index: 47. (index mismatch)
I20250113 21:10:04.134719 28438 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250113 21:10:04.212266 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:04.235352 28434 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:04.235453 28434 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 13 FOLLOWER]: Advancing to term 14
I20250113 21:10:04.235657 28425 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:04.235718 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 13 FOLLOWER]: Advancing to term 14
I20250113 21:10:04.236197 28434 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.236254 28425 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.236353 28434 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 14 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:04.236523 28425 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 14 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:04.236668 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:04.236529 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:04.236541 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:04.236748 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 14: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:04.236565 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:04.236814 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 14: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:04.236943 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:04.237010 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:04.237023 28425 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250113 21:10:04.237094 28434 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250113 21:10:04.241497 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 14 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 13. Current term is 14. Ops: []
I20250113 21:10:04.241691 28438 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 13.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250113 21:10:04.241792 28438 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 13 LEADER]: Stepping down as leader of term 13
I20250113 21:10:04.241825 28438 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:04.241901 28438 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 13.47, Last appended by leader: 47, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.241976 28438 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 13 FOLLOWER]: Advancing to term 14
I20250113 21:10:04.241999 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 14 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 13. Current term is 14. Ops: []
I20250113 21:10:04.354498 28425 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:04.354589 28425 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 14 FOLLOWER]: Advancing to term 15
I20250113 21:10:04.355224 28425 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.355374 28425 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 15 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:04.355607 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 15 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:04.355677 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 14 FOLLOWER]: Advancing to term 15
I20250113 21:10:04.355614 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 15 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:04.355728 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 14 FOLLOWER]: Advancing to term 15
I20250113 21:10:04.356223 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 15.
I20250113 21:10:04.356309 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 15.
I20250113 21:10:04.356421 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters:
I20250113 21:10:04.356531 28425 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 15 FOLLOWER]: Leader election won for term 15
I20250113 21:10:04.356586 28425 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 15 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:04.356637 28425 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 13.47, Last appended by leader: 47, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.357044 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 13 to 15, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 15 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:04.443418 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 15 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 13 index: 47. Preceding OpId from leader: term: 15 index: 48. (index mismatch)
I20250113 21:10:04.443691 28425 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250113 21:10:04.445015 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 15 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 13 index: 47. Preceding OpId from leader: term: 15 index: 48. (index mismatch)
I20250113 21:10:04.445225 28436 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250113 21:10:04.724575 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:04.739418 28434 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:04.739491 28434 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 15 FOLLOWER]: Advancing to term 16
I20250113 21:10:04.740115 28434 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.740260 28434 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 16 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:04.740437 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:04.740468 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:04.740694 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:04.740803 28434 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250113 21:10:04.757134 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 16 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 15. Current term is 16. Ops: []
I20250113 21:10:04.757351 28440 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 15.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250113 21:10:04.757467 28440 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 15 LEADER]: Stepping down as leader of term 15
I20250113 21:10:04.757500 28440 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:04.757551 28440 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 15.48, Last appended by leader: 48, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.757669 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 15 FOLLOWER]: Advancing to term 16
I20250113 21:10:04.816886 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:04.816994 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 15 FOLLOWER]: Advancing to term 16
I20250113 21:10:04.817832 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.818032 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 16 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:04.818244 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:04.818327 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 16: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:04.818249 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:04.818946 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 16.
I20250113 21:10:04.819129 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:04.819219 28433 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 16 FOLLOWER]: Leader election won for term 16
I20250113 21:10:04.819272 28433 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 16 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:04.819310 28433 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 15.48, Last appended by leader: 48, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.819778 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 15 to 16, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 16 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:04.841256 28434 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:04.841336 28434 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 16 FOLLOWER]: Advancing to term 17
I20250113 21:10:04.841971 28434 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.842139 28434 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 17 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:04.842332 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 17 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:04.842340 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 17 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:04.842430 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 16 FOLLOWER]: Advancing to term 17
I20250113 21:10:04.842979 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 17.
I20250113 21:10:04.843142 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:04.843223 28434 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 17 FOLLOWER]: Leader election won for term 17
I20250113 21:10:04.843325 28434 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 17 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:04.843365 28434 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 15.48, Last appended by leader: 48, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.843794 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 16 to 17, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 17 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:04.907231 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 17 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 16. Current term is 17. Ops: []
I20250113 21:10:04.907505 28433 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250113 21:10:04.907636 28433 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 17.49 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 48
I20250113 21:10:04.907670 28433 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 16 LEADER]: Stepping down as leader of term 16
I20250113 21:10:04.907697 28433 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:04.907748 28433 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 16.49, Last appended by leader: 49, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:04.907821 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 16 FOLLOWER]: Advancing to term 17
I20250113 21:10:04.950271 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 17 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 15 index: 48. Preceding OpId from leader: term: 17 index: 49. (index mismatch)
I20250113 21:10:04.950553 28434 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250113 21:10:04.951545 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 17 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 16 index: 49. Preceding OpId from leader: term: 17 index: 49. (term mismatch)
I20250113 21:10:04.951597 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 48
I20250113 21:10:04.951618 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 16.49
I20250113 21:10:04.951648 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:04.951859 28442 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250113 21:10:05.229514 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:05.229621 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 17 FOLLOWER]: Advancing to term 18
I20250113 21:10:05.230320 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.230485 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 18 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:05.230700 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:05.230696 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:05.230921 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:05.231019 28433 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20250113 21:10:05.232367 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 18 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 17. Current term is 18. Ops: []
I20250113 21:10:05.232574 28434 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 17.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250113 21:10:05.232668 28434 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 17 LEADER]: Stepping down as leader of term 17
I20250113 21:10:05.232710 28434 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:05.232760 28434 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 17.49, Last appended by leader: 49, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.232839 28434 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 17 FOLLOWER]: Advancing to term 18
I20250113 21:10:05.240042 28440 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:05.240101 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 17 FOLLOWER]: Advancing to term 18
I20250113 21:10:05.240598 28440 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.240729 28440 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 18 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:05.240903 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:05.240922 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:05.240989 28180 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 18: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:05.241521 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 18.
I20250113 21:10:05.241693 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:05.241786 28440 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 18 FOLLOWER]: Leader election won for term 18
I20250113 21:10:05.241879 28440 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 18 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:05.241937 28440 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 17.49, Last appended by leader: 49, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.242358 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 17 to 18, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 18 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:05.318774 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 18 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 17 index: 49. Preceding OpId from leader: term: 18 index: 50. (index mismatch)
I20250113 21:10:05.319037 28440 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250113 21:10:05.320000 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 18 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 17 index: 49. Preceding OpId from leader: term: 18 index: 50. (index mismatch)
I20250113 21:10:05.320384 28425 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250113 21:10:05.597328 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:05.597430 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 18 FOLLOWER]: Advancing to term 19
I20250113 21:10:05.598137 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.598284 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 19 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:05.598500 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 19 candidate_status { last_received { term: 18 index: 50 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:05.598505 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 19 candidate_status { last_received { term: 18 index: 50 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:05.598762 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:05.598903 28433 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250113 21:10:05.608702 28434 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:05.608760 28434 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 18 FOLLOWER]: Advancing to term 19
I20250113 21:10:05.609323 28434 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.609453 28434 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 19 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:05.609637 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 19 candidate_status { last_received { term: 18 index: 50 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:05.609634 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 19 candidate_status { last_received { term: 18 index: 50 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:05.609748 28180 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 19: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:05.609911 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:05.610024 28434 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250113 21:10:05.611147 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 19 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 18. Current term is 19. Ops: []
I20250113 21:10:05.611331 28443 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 18.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250113 21:10:05.611430 28440 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 18 LEADER]: Stepping down as leader of term 18
I20250113 21:10:05.611469 28440 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:05.611514 28440 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 18.50, Last appended by leader: 50, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.611582 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 18 FOLLOWER]: Advancing to term 19
I20250113 21:10:05.715378 28434 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:05.715471 28434 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 19 FOLLOWER]: Advancing to term 20
I20250113 21:10:05.716084 28434 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.716235 28434 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 20 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:05.716450 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 20 candidate_status { last_received { term: 18 index: 50 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:05.716450 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 20 candidate_status { last_received { term: 18 index: 50 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:05.716552 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 19 FOLLOWER]: Advancing to term 20
I20250113 21:10:05.716558 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 19 FOLLOWER]: Advancing to term 20
I20250113 21:10:05.717212 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 20.
I20250113 21:10:05.717212 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 20.
I20250113 21:10:05.717456 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:05.717566 28434 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 20 FOLLOWER]: Leader election won for term 20
I20250113 21:10:05.717645 28434 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 20 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:05.717798 28434 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 18.50, Last appended by leader: 50, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.718253 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 18 to 20, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 20 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:05.724800 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:05.801779 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 20 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 18 index: 50. Preceding OpId from leader: term: 20 index: 51. (index mismatch)
I20250113 21:10:05.802080 28434 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250113 21:10:05.803084 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 20 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 18 index: 50. Preceding OpId from leader: term: 20 index: 51. (index mismatch)
I20250113 21:10:05.803242 28441 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250113 21:10:05.904058 28440 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:05.904160 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 20 FOLLOWER]: Advancing to term 21
I20250113 21:10:05.904932 28440 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.905071 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 21 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 20. Current term is 21. Ops: []
I20250113 21:10:05.905088 28440 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 21 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:05.905244 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:05.905339 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:05.905521 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:05.905601 28444 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 20.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250113 21:10:05.905622 28440 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20250113 21:10:05.905697 28444 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 20 LEADER]: Stepping down as leader of term 20
I20250113 21:10:05.905731 28444 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:05.905784 28444 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 20.51, Last appended by leader: 51, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.905856 28444 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 20 FOLLOWER]: Advancing to term 21
I20250113 21:10:05.991022 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:05.991125 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 20 FOLLOWER]: Advancing to term 21
I20250113 21:10:05.991861 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.992040 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 21 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:05.992262 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:05.992347 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 21: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:05.992257 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:05.993037 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 21.
I20250113 21:10:05.993222 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:05.993314 28433 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 21 FOLLOWER]: Leader election won for term 21
I20250113 21:10:05.993440 28433 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 21 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:05.993499 28433 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 20.51, Last appended by leader: 51, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:05.994021 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 20 to 21, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 21 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:06.023097 28440 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:06.023198 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 21 FOLLOWER]: Advancing to term 22
I20250113 21:10:06.023854 28440 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.024035 28440 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 22 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:06.024277 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 22 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:06.024348 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 21 FOLLOWER]: Advancing to term 22
I20250113 21:10:06.024281 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 22 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:06.024921 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 22.
I20250113 21:10:06.025072 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:06.025168 28440 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 22 FOLLOWER]: Leader election won for term 22
I20250113 21:10:06.025228 28440 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 22 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:06.025282 28440 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 20.51, Last appended by leader: 51, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.025741 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 21 to 22, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 22 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:06.079449 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 22 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 21. Current term is 22. Ops: []
I20250113 21:10:06.079784 28433 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250113 21:10:06.079881 28433 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 22.52 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I20250113 21:10:06.079903 28433 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 21 LEADER]: Stepping down as leader of term 21
I20250113 21:10:06.079923 28433 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:06.079983 28433 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 21.52, Last appended by leader: 52, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.080049 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 21 FOLLOWER]: Advancing to term 22
I20250113 21:10:06.132305 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 22 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 20 index: 51. Preceding OpId from leader: term: 22 index: 52. (index mismatch)
I20250113 21:10:06.132548 28440 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250113 21:10:06.133512 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 22 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 21 index: 52. Preceding OpId from leader: term: 22 index: 52. (term mismatch)
I20250113 21:10:06.133571 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 51
I20250113 21:10:06.133594 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 21.52
I20250113 21:10:06.133611 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:06.133836 28443 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250113 21:10:06.502395 28444 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:06.502494 28444 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 22 FOLLOWER]: Advancing to term 23
I20250113 21:10:06.503365 28444 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.503504 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 23 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 22. Current term is 23. Ops: []
I20250113 21:10:06.503656 28444 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 23 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:06.503691 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:06.503674 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:06.503700 28446 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 22.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250113 21:10:06.503921 28440 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 22 LEADER]: Stepping down as leader of term 22
I20250113 21:10:06.503962 28440 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:06.503970 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:06.504034 28440 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 22.52, Last appended by leader: 52, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.504066 28444 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20250113 21:10:06.504129 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 22 FOLLOWER]: Advancing to term 23
I20250113 21:10:06.519155 28445 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:06.519219 28445 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 22 FOLLOWER]: Advancing to term 23
I20250113 21:10:06.519793 28445 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.519932 28445 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 23 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:06.520109 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:06.520128 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:06.520195 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 23: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:06.520709 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 23.
I20250113 21:10:06.520887 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:06.520989 28445 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 23 FOLLOWER]: Leader election won for term 23
I20250113 21:10:06.521064 28445 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 23 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:06.521122 28445 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 22.52, Last appended by leader: 52, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.521564 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 22 to 23, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 23 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:06.612149 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 23 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 22 index: 52. Preceding OpId from leader: term: 23 index: 53. (index mismatch)
I20250113 21:10:06.612414 28445 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250113 21:10:06.613430 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 23 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 22 index: 52. Preceding OpId from leader: term: 23 index: 53. (index mismatch)
I20250113 21:10:06.613608 28445 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250113 21:10:06.714623 28447 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:06.714743 28447 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 23 FOLLOWER]: Advancing to term 24
I20250113 21:10:06.715780 28447 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.716004 28447 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 24 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:06.716218 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 24 candidate_status { last_received { term: 23 index: 53 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:06.716239 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 24 candidate_status { last_received { term: 23 index: 53 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:06.716320 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 23 FOLLOWER]: Advancing to term 24
I20250113 21:10:06.716862 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 24.
I20250113 21:10:06.717056 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:06.717183 28447 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 24 FOLLOWER]: Leader election won for term 24
I20250113 21:10:06.717271 28447 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 24 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:06.717350 28447 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 23.53, Last appended by leader: 53, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.717979 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 23 to 24, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 24 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:06.725052 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:06.727041 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 24 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 23. Current term is 24. Ops: []
I20250113 21:10:06.727241 28448 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 23.53, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250113 21:10:06.727331 28448 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 24.54 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 53
I20250113 21:10:06.727387 28433 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 23 LEADER]: Stepping down as leader of term 23
I20250113 21:10:06.727452 28433 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:06.727505 28433 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 23.53, Last appended by leader: 53, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.727582 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 23 FOLLOWER]: Advancing to term 24
I20250113 21:10:06.813344 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 24 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 23 index: 53. Preceding OpId from leader: term: 24 index: 54. (index mismatch)
I20250113 21:10:06.813648 28447 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250113 21:10:06.814620 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 24 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 23 index: 53. Preceding OpId from leader: term: 24 index: 54. (index mismatch)
I20250113 21:10:06.814795 28447 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250113 21:10:06.915153 28440 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:06.915259 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 24 FOLLOWER]: Advancing to term 25
I20250113 21:10:06.915922 28440 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.916080 28440 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 25 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:06.916286 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 25 candidate_status { last_received { term: 24 index: 54 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:06.916349 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 24 FOLLOWER]: Advancing to term 25
I20250113 21:10:06.916327 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 25 candidate_status { last_received { term: 24 index: 54 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:06.916937 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 25.
I20250113 21:10:06.917106 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:06.917183 28440 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 25 FOLLOWER]: Leader election won for term 25
I20250113 21:10:06.917235 28440 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 25 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:06.917290 28440 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 24.54, Last appended by leader: 54, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.917708 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 24 to 25, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 25 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:06.935802 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 25 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 24. Current term is 25. Ops: []
I20250113 21:10:06.936003 28444 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 24.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250113 21:10:06.936107 28444 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 24 LEADER]: Stepping down as leader of term 24
I20250113 21:10:06.936147 28444 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:06.936201 28444 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 24.54, Last appended by leader: 54, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:06.936290 28444 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 24 FOLLOWER]: Advancing to term 25
I20250113 21:10:07.020362 28433 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:07.020459 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 25 FOLLOWER]: Advancing to term 26
I20250113 21:10:07.021126 28433 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.021293 28433 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 26 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:07.021515 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 26 candidate_status { last_received { term: 24 index: 54 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:07.021499 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 26 candidate_status { last_received { term: 24 index: 54 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:07.021638 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 25 FOLLOWER]: Advancing to term 26
I20250113 21:10:07.022301 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 26.
I20250113 21:10:07.022469 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:07.022545 28433 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 26 FOLLOWER]: Leader election won for term 26
I20250113 21:10:07.022672 28433 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 26 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:07.022823 28433 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 24.54, Last appended by leader: 54, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.022917 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 26 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 25. Current term is 26. Ops: []
I20250113 21:10:07.023102 28440 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250113 21:10:07.023198 28440 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 25 LEADER]: Stepping down as leader of term 25
I20250113 21:10:07.023234 28440 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:07.023289 28440 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 25.55, Last appended by leader: 55, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.023352 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 25 FOLLOWER]: Advancing to term 26
I20250113 21:10:07.023319 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 25 to 26, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 26 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:07.104784 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 26 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 24 index: 54. Preceding OpId from leader: term: 26 index: 55. (index mismatch)
I20250113 21:10:07.105074 28433 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250113 21:10:07.106034 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 26 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 25 index: 55. Preceding OpId from leader: term: 26 index: 55. (term mismatch)
I20250113 21:10:07.106087 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 54
I20250113 21:10:07.106108 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 25.55
I20250113 21:10:07.106132 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:07.106313 28448 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250113 21:10:07.284173 28440 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:07.284277 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 26 FOLLOWER]: Advancing to term 27
I20250113 21:10:07.284927 28440 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.285091 28440 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 27 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:07.285292 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 27 candidate_status { last_received { term: 26 index: 55 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:07.285315 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 27 candidate_status { last_received { term: 26 index: 55 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:07.285502 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:07.285624 28440 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20250113 21:10:07.291698 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 27 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 26. Current term is 27. Ops: []
I20250113 21:10:07.291909 28433 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 26.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250113 21:10:07.292006 28433 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 26 LEADER]: Stepping down as leader of term 26
I20250113 21:10:07.292042 28433 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:07.292084 28433 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 26.55, Last appended by leader: 55, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.292176 28433 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 26 FOLLOWER]: Advancing to term 27
I20250113 21:10:07.295603 28444 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:07.295663 28444 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 26 FOLLOWER]: Advancing to term 27
I20250113 21:10:07.296471 28444 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.296638 28444 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 27 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:07.296784 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 27 candidate_status { last_received { term: 26 index: 55 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:07.296831 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 27 candidate_status { last_received { term: 26 index: 55 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:07.296976 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 27: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:07.297545 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 27.
I20250113 21:10:07.297734 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:07.297894 28444 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 27 FOLLOWER]: Leader election won for term 27
I20250113 21:10:07.297955 28444 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 27 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:07.298007 28444 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 26.55, Last appended by leader: 55, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.298504 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 26 to 27, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 27 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:07.373782 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 27 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 26 index: 55. Preceding OpId from leader: term: 27 index: 56. (index mismatch)
I20250113 21:10:07.374068 28447 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250113 21:10:07.375043 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 27 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 26 index: 55. Preceding OpId from leader: term: 27 index: 56. (index mismatch)
I20250113 21:10:07.375226 28450 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250113 21:10:07.476260 28446 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:07.476382 28446 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 27 FOLLOWER]: Advancing to term 28
I20250113 21:10:07.477135 28446 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.477312 28446 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 28 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:07.477515 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 28 candidate_status { last_received { term: 27 index: 56 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:07.477530 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 28 candidate_status { last_received { term: 27 index: 56 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:07.477634 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 27 FOLLOWER]: Advancing to term 28
I20250113 21:10:07.478216 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 28.
I20250113 21:10:07.478374 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:07.478456 28446 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 28 FOLLOWER]: Leader election won for term 28
I20250113 21:10:07.478509 28446 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 28 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:07.478551 28446 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 27.56, Last appended by leader: 56, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.479013 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 27 to 28, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 28 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:07.480077 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 28 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 27. Current term is 28. Ops: []
I20250113 21:10:07.480266 28447 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 27.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250113 21:10:07.480353 28447 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 28.57 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 56
I20250113 21:10:07.480392 28450 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 27 LEADER]: Stepping down as leader of term 27
I20250113 21:10:07.480437 28450 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:07.480494 28450 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 27.56, Last appended by leader: 56, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.480569 28450 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 27 FOLLOWER]: Advancing to term 28
I20250113 21:10:07.574685 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 28 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 27 index: 56. Preceding OpId from leader: term: 28 index: 57. (index mismatch)
I20250113 21:10:07.574959 28446 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250113 21:10:07.575917 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 28 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 27 index: 56. Preceding OpId from leader: term: 28 index: 57. (index mismatch)
I20250113 21:10:07.576082 28440 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250113 21:10:07.725247 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:07.857635 28450 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:07.857739 28450 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 28 FOLLOWER]: Advancing to term 29
I20250113 21:10:07.858400 28450 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.858541 28450 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 29 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:07.858745 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 29 candidate_status { last_received { term: 28 index: 57 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:07.858745 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 29 candidate_status { last_received { term: 28 index: 57 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:07.859047 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:07.859122 28450 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20250113 21:10:07.860369 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 29 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 28. Current term is 29. Ops: []
I20250113 21:10:07.860561 28440 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 28.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250113 21:10:07.860642 28440 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 28 LEADER]: Stepping down as leader of term 28
I20250113 21:10:07.860675 28440 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:07.860723 28440 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 28.57, Last appended by leader: 57, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.860800 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 28 FOLLOWER]: Advancing to term 29
I20250113 21:10:07.939267 28448 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:07.939357 28448 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 28 FOLLOWER]: Advancing to term 29
I20250113 21:10:07.940150 28448 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.940325 28448 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 29 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:07.940529 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 29 candidate_status { last_received { term: 28 index: 57 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:07.940534 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 29 candidate_status { last_received { term: 28 index: 57 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:07.940605 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 29: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:07.941162 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 29.
I20250113 21:10:07.941339 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:07.941427 28448 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 29 FOLLOWER]: Leader election won for term 29
I20250113 21:10:07.941481 28448 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 29 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:07.941531 28448 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 28.57, Last appended by leader: 57, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:07.941975 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 28 to 29, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 29 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:08.002643 28450 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:08.002744 28450 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 29 FOLLOWER]: Advancing to term 30
I20250113 21:10:08.003386 28450 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.003544 28450 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 30 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:08.003790 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 30 candidate_status { last_received { term: 28 index: 57 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:08.003798 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 30 candidate_status { last_received { term: 28 index: 57 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:08.003892 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 29 FOLLOWER]: Advancing to term 30
I20250113 21:10:08.004426 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 30.
I20250113 21:10:08.004602 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:08.004690 28450 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 30 FOLLOWER]: Leader election won for term 30
I20250113 21:10:08.004837 28450 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 30 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:08.004887 28450 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 28.57, Last appended by leader: 57, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.005309 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 29 to 30, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 30 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:08.044454 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 30 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 29. Current term is 30. Ops: []
I20250113 21:10:08.044720 28448 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250113 21:10:08.044840 28448 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 29 LEADER]: Stepping down as leader of term 29
I20250113 21:10:08.044876 28448 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:08.044934 28448 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 29.58, Last appended by leader: 58, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.045006 28448 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 29 FOLLOWER]: Advancing to term 30
I20250113 21:10:08.088950 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 30 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 28 index: 57. Preceding OpId from leader: term: 30 index: 58. (index mismatch)
I20250113 21:10:08.089228 28450 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250113 21:10:08.090230 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 30 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 29 index: 58. Preceding OpId from leader: term: 30 index: 58. (term mismatch)
I20250113 21:10:08.090287 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 57
I20250113 21:10:08.090305 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 29.58
I20250113 21:10:08.090323 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:08.090497 28451 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250113 21:10:08.191241 28440 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:08.191365 28440 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 30 FOLLOWER]: Advancing to term 31
I20250113 21:10:08.192080 28440 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.192236 28440 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 31 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:08.192454 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 31 candidate_status { last_received { term: 30 index: 58 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:08.192459 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 31 candidate_status { last_received { term: 30 index: 58 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:08.192562 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 30 FOLLOWER]: Advancing to term 31
I20250113 21:10:08.193073 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 31.
I20250113 21:10:08.193240 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:08.193334 28440 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 31 FOLLOWER]: Leader election won for term 31
I20250113 21:10:08.193395 28440 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 31 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:08.193449 28440 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 30.58, Last appended by leader: 58, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.193907 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 30 to 31, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 31 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:08.200186 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 31 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 30. Current term is 31. Ops: []
I20250113 21:10:08.200387 28451 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 30.58, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250113 21:10:08.200456 28451 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 31.59 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 58
I20250113 21:10:08.200487 28450 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 30 LEADER]: Stepping down as leader of term 30
I20250113 21:10:08.200537 28450 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:08.200600 28450 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 30.58, Last appended by leader: 58, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.200683 28450 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 30 FOLLOWER]: Advancing to term 31
I20250113 21:10:08.291548 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 31 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 30 index: 58. Preceding OpId from leader: term: 31 index: 59. (index mismatch)
I20250113 21:10:08.291823 28440 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250113 21:10:08.292773 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 31 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 30 index: 58. Preceding OpId from leader: term: 31 index: 59. (index mismatch)
I20250113 21:10:08.292982 28452 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250113 21:10:08.514606 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 31 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:08.514699 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 31 FOLLOWER]: Advancing to term 32
I20250113 21:10:08.515416 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.515544 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 32 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 31. Current term is 32. Ops: []
I20250113 21:10:08.515596 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 32 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:08.515750 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 32 candidate_status { last_received { term: 31 index: 59 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:08.515774 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 32 candidate_status { last_received { term: 31 index: 59 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:08.515808 28446 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 31.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250113 21:10:08.515911 28446 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 31 LEADER]: Stepping down as leader of term 31
I20250113 21:10:08.515950 28446 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:08.516013 28446 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 31.59, Last appended by leader: 59, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.516089 28446 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 31 FOLLOWER]: Advancing to term 32
I20250113 21:10:08.516662 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:08.516793 28451 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250113 21:10:08.579340 28448 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:08.579435 28448 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 31 FOLLOWER]: Advancing to term 32
I20250113 21:10:08.580086 28448 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.580245 28448 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 32 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:08.580457 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 32 candidate_status { last_received { term: 31 index: 59 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:08.580526 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 32: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:08.580461 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 32 candidate_status { last_received { term: 31 index: 59 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:08.581072 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 32.
I20250113 21:10:08.581243 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:08.581331 28448 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 32 FOLLOWER]: Leader election won for term 32
I20250113 21:10:08.581446 28448 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 32 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:08.581501 28448 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 31.59, Last appended by leader: 59, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.581956 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 31 to 32, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 32 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:08.664247 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:08.664347 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 32 FOLLOWER]: Advancing to term 33
I20250113 21:10:08.665053 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.665223 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 33 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:08.665443 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 33 candidate_status { last_received { term: 31 index: 59 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:08.665443 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 33 candidate_status { last_received { term: 31 index: 59 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:08.665576 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 32 FOLLOWER]: Advancing to term 33
I20250113 21:10:08.666204 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 33.
I20250113 21:10:08.666360 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:08.666445 28451 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 33 FOLLOWER]: Leader election won for term 33
I20250113 21:10:08.666522 28451 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 33 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:08.666582 28451 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 31.59, Last appended by leader: 59, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.667064 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 32 to 33, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 33 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:08.674171 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 33 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 32. Current term is 33. Ops: []
I20250113 21:10:08.674350 28448 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250113 21:10:08.674427 28448 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 33.60 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 59
I20250113 21:10:08.674456 28453 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 32 LEADER]: Stepping down as leader of term 32
I20250113 21:10:08.674503 28453 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:08.674567 28453 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 32.60, Last appended by leader: 60, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.674647 28453 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 32 FOLLOWER]: Advancing to term 33
I20250113 21:10:08.725587 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:08.751448 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 33 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 31 index: 59. Preceding OpId from leader: term: 33 index: 60. (index mismatch)
I20250113 21:10:08.751709 28451 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250113 21:10:08.752607 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 33 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 32 index: 60. Preceding OpId from leader: term: 33 index: 60. (term mismatch)
I20250113 21:10:08.752658 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 59
I20250113 21:10:08.752694 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 32.60
I20250113 21:10:08.752717 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:08.752928 28450 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250113 21:10:08.852988 28446 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:08.853092 28446 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 33 FOLLOWER]: Advancing to term 34
I20250113 21:10:08.853847 28446 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.854066 28446 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 34 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:08.854291 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 34 candidate_status { last_received { term: 33 index: 60 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:08.854298 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 34 candidate_status { last_received { term: 33 index: 60 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:08.854404 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 33 FOLLOWER]: Advancing to term 34
I20250113 21:10:08.854944 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 34.
I20250113 21:10:08.855103 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:08.855180 28446 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 34 FOLLOWER]: Leader election won for term 34
I20250113 21:10:08.855242 28446 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 34 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:08.855297 28446 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 33.60, Last appended by leader: 60, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.855742 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 33 to 34, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 34 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:08.865705 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 34 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 33. Current term is 34. Ops: []
I20250113 21:10:08.865810 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 34 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 33. Current term is 34. Ops: []
I20250113 21:10:08.865935 28454 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 33.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250113 21:10:08.866065 28454 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 33 LEADER]: Stepping down as leader of term 33
I20250113 21:10:08.866111 28454 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:08.866076 28451 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 33.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250113 21:10:08.866182 28451 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 34.61 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 60
I20250113 21:10:08.866228 28454 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 33.60, Last appended by leader: 60, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:08.866300 28454 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 33 FOLLOWER]: Advancing to term 34
I20250113 21:10:08.940091 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 34 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 33 index: 60. Preceding OpId from leader: term: 34 index: 61. (index mismatch)
I20250113 21:10:08.940340 28446 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250113 21:10:08.941538 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 34 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 33 index: 60. Preceding OpId from leader: term: 34 index: 61. (index mismatch)
I20250113 21:10:08.941722 28455 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250113 21:10:09.042451 28453 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 34 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:09.042550 28453 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 34 FOLLOWER]: Advancing to term 35
I20250113 21:10:09.043262 28453 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.043422 28453 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 35 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:09.043638 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 35 candidate_status { last_received { term: 34 index: 61 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:09.043649 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 35 candidate_status { last_received { term: 34 index: 61 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:09.043767 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 34 FOLLOWER]: Advancing to term 35
I20250113 21:10:09.044314 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 35.
I20250113 21:10:09.044495 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:09.044584 28453 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 35 FOLLOWER]: Leader election won for term 35
I20250113 21:10:09.044648 28453 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 35 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:09.044700 28453 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 34.61, Last appended by leader: 61, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.045115 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 34 to 35, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 35 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:09.062111 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 35 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 34. Current term is 35. Ops: []
I20250113 21:10:09.062330 28455 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 34.61, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250113 21:10:09.062418 28455 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 35.62 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 61
I20250113 21:10:09.062463 28446 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 34 LEADER]: Stepping down as leader of term 34
I20250113 21:10:09.062508 28446 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:09.062569 28446 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 34.61, Last appended by leader: 61, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.062647 28446 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 34 FOLLOWER]: Advancing to term 35
I20250113 21:10:09.139637 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 35 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 34 index: 61. Preceding OpId from leader: term: 35 index: 62. (index mismatch)
I20250113 21:10:09.139886 28453 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250113 21:10:09.140851 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 35 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 34 index: 61. Preceding OpId from leader: term: 35 index: 62. (index mismatch)
I20250113 21:10:09.141091 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250113 21:10:09.241286 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 35 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:09.241380 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 35 FOLLOWER]: Advancing to term 36
I20250113 21:10:09.242077 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.242236 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 36 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:09.242430 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 36 candidate_status { last_received { term: 35 index: 62 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:09.242430 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 36 candidate_status { last_received { term: 35 index: 62 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:09.242699 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:09.242798 28451 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20250113 21:10:09.264050 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 36 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 35. Current term is 36. Ops: []
I20250113 21:10:09.264302 28453 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 35.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250113 21:10:09.264459 28456 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 35 LEADER]: Stepping down as leader of term 35
I20250113 21:10:09.264523 28456 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:09.264591 28456 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 35.62, Last appended by leader: 62, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.264714 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 35 FOLLOWER]: Advancing to term 36
I20250113 21:10:09.325423 28455 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:09.325527 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 35 FOLLOWER]: Advancing to term 36
I20250113 21:10:09.326273 28455 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.326422 28455 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 36 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:09.326639 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 36 candidate_status { last_received { term: 35 index: 62 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:09.326715 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 36: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:09.326663 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 36 candidate_status { last_received { term: 35 index: 62 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:09.327257 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 36.
I20250113 21:10:09.327417 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:09.327502 28455 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 36 FOLLOWER]: Leader election won for term 36
I20250113 21:10:09.327571 28455 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 36 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:09.327672 28455 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 35.62, Last appended by leader: 62, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.328158 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 35 to 36, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 36 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:09.368284 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:09.368381 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 36 FOLLOWER]: Advancing to term 37
I20250113 21:10:09.369005 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.369158 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 37 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:09.369377 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 37 candidate_status { last_received { term: 35 index: 62 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:09.369377 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 37 candidate_status { last_received { term: 35 index: 62 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:09.369530 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 36 FOLLOWER]: Advancing to term 37
I20250113 21:10:09.370110 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 37.
I20250113 21:10:09.370272 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:09.370352 28451 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 37 FOLLOWER]: Leader election won for term 37
I20250113 21:10:09.370414 28451 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 37 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:09.370474 28451 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 35.62, Last appended by leader: 62, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.370937 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 36 to 37, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 37 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:09.415715 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 37 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 36. Current term is 37. Ops: []
I20250113 21:10:09.416025 28455 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250113 21:10:09.416131 28455 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 36 LEADER]: Stepping down as leader of term 36
I20250113 21:10:09.416148 28455 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:09.416198 28455 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 36.63, Last appended by leader: 63, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.416268 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 36 FOLLOWER]: Advancing to term 37
I20250113 21:10:09.448532 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 37 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 36 index: 63. Preceding OpId from leader: term: 37 index: 63. (term mismatch)
I20250113 21:10:09.448613 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 62
I20250113 21:10:09.448628 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 36.63
I20250113 21:10:09.448645 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:09.448863 28451 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250113 21:10:09.450898 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 37 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 35 index: 62. Preceding OpId from leader: term: 37 index: 63. (index mismatch)
I20250113 21:10:09.451136 28454 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250113 21:10:09.633944 28453 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 37 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:09.634040 28453 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 37 FOLLOWER]: Advancing to term 38
I20250113 21:10:09.634688 28453 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.634892 28453 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 38 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:09.635129 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 38 candidate_status { last_received { term: 37 index: 63 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:09.635210 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 38 candidate_status { last_received { term: 37 index: 63 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:09.635448 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:09.635545 28453 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20250113 21:10:09.654186 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 38 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 37. Current term is 38. Ops: []
I20250113 21:10:09.654385 28451 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 37.63, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250113 21:10:09.654497 28451 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 37 LEADER]: Stepping down as leader of term 37
I20250113 21:10:09.654536 28451 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:09.654592 28451 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 37.63, Last appended by leader: 63, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.654670 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 37 FOLLOWER]: Advancing to term 38
I20250113 21:10:09.725852 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:09.745939 28453 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:09.746024 28453 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 38 FOLLOWER]: Advancing to term 39
I20250113 21:10:09.746704 28453 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.746865 28453 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 39 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:09.747069 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 39 candidate_status { last_received { term: 37 index: 63 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:09.747069 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 39 candidate_status { last_received { term: 37 index: 63 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:09.747174 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 38 FOLLOWER]: Advancing to term 39
I20250113 21:10:09.747731 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 39.
I20250113 21:10:09.747898 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:09.748006 28453 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 39 FOLLOWER]: Leader election won for term 39
I20250113 21:10:09.748067 28453 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 39 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:09.748121 28453 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 37.63, Last appended by leader: 63, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.748593 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 37 to 39, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 39 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:09.750375 28455 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:09.750422 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 37 FOLLOWER]: Advancing to term 38
I20250113 21:10:09.750914 28455 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.751037 28455 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 38 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:09.751180 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 38 candidate_status { last_received { term: 37 index: 63 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:09.751272 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 38 candidate_status { last_received { term: 37 index: 63 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:09.751319 28256 raft_consensus.cc:2363] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for earlier term 38. Current term is 39.
I20250113 21:10:09.751437 28295 leader_election.cc:400] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 38 election: Vote denied by peer 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759) with higher term. Message: Invalid argument: T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for earlier term 38. Current term is 39.
I20250113 21:10:09.751468 28295 leader_election.cc:403] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20250113 21:10:09.751516 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 38 FOLLOWER]: Advancing to term 39
I20250113 21:10:09.752012 28455 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759) with higher term. Message: Invalid argument: T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for earlier term 38. Current term is 39.
I20250113 21:10:09.861153 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 39 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 37 index: 63. Preceding OpId from leader: term: 39 index: 64. (index mismatch)
I20250113 21:10:09.861421 28453 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250113 21:10:09.862105 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 39 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 37 index: 63. Preceding OpId from leader: term: 39 index: 64. (index mismatch)
I20250113 21:10:09.862318 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250113 21:10:09.963291 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 39 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:09.963394 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 39 FOLLOWER]: Advancing to term 40
I20250113 21:10:09.963656 28455 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:09.963714 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 39 FOLLOWER]: Advancing to term 40
I20250113 21:10:09.964208 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.964332 28455 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.964561 28455 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 40 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:09.964610 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 40 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:09.964643 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 40 candidate_status { last_received { term: 39 index: 64 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:09.964675 28331 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 40 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 39. Current term is 40. Ops: []
I20250113 21:10:09.964730 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 40 candidate_status { last_received { term: 39 index: 64 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:09.964576 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 40 candidate_status { last_received { term: 39 index: 64 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:09.964794 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 40: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:09.964793 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 40: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:09.964862 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 40 candidate_status { last_received { term: 39 index: 64 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:09.964982 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:09.965088 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:09.964975 28456 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 39.64, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250113 21:10:09.965113 28455 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20250113 21:10:09.965142 28456 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 39 LEADER]: Stepping down as leader of term 39
I20250113 21:10:09.965168 28456 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:09.965190 28451 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20250113 21:10:09.965317 28456 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 39.64, Last appended by leader: 64, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:09.965389 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 39 FOLLOWER]: Advancing to term 40
I20250113 21:10:10.101629 28455 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:10.101727 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 40 FOLLOWER]: Advancing to term 41
I20250113 21:10:10.102419 28455 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.102608 28455 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 41 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:10.102828 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 41 candidate_status { last_received { term: 39 index: 64 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:10.102828 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 41 candidate_status { last_received { term: 39 index: 64 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:10.102926 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 40 FOLLOWER]: Advancing to term 41
I20250113 21:10:10.102931 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 40 FOLLOWER]: Advancing to term 41
I20250113 21:10:10.103622 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 41.
I20250113 21:10:10.103704 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 41.
I20250113 21:10:10.103814 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:10.103897 28455 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 41 FOLLOWER]: Leader election won for term 41
I20250113 21:10:10.103947 28455 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 41 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:10.104005 28455 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 39.64, Last appended by leader: 64, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.104521 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 39 to 41, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 41 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:10.186002 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 41 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 39 index: 64. Preceding OpId from leader: term: 41 index: 65. (index mismatch)
I20250113 21:10:10.186265 28455 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250113 21:10:10.187189 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 41 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 39 index: 64. Preceding OpId from leader: term: 41 index: 65. (index mismatch)
I20250113 21:10:10.187414 28446 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250113 21:10:10.545295 28456 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 41 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:10.545375 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 41 FOLLOWER]: Advancing to term 42
I20250113 21:10:10.546072 28456 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.546247 28456 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 42 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:10.546448 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 42 candidate_status { last_received { term: 41 index: 65 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:10.546458 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 42 candidate_status { last_received { term: 41 index: 65 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:10.546762 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:10.546870 28456 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20250113 21:10:10.555485 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 42 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 41. Current term is 42. Ops: []
I20250113 21:10:10.555694 28460 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 41.65, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250113 21:10:10.555794 28460 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 41 LEADER]: Stepping down as leader of term 41
I20250113 21:10:10.555824 28460 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:10.555886 28460 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 41.65, Last appended by leader: 65, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.555966 28460 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 41 FOLLOWER]: Advancing to term 42
I20250113 21:10:10.588603 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 41 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:10.588686 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 41 FOLLOWER]: Advancing to term 42
I20250113 21:10:10.589331 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.589471 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 42 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:10.589727 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 42 candidate_status { last_received { term: 41 index: 65 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:10.589787 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 42 candidate_status { last_received { term: 41 index: 65 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:10.589897 28180 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 42: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:10.590405 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 42.
I20250113 21:10:10.590574 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:10.590675 28451 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 42 FOLLOWER]: Leader election won for term 42
I20250113 21:10:10.590775 28451 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 42 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:10.590834 28451 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 41.65, Last appended by leader: 65, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.591245 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 41 to 42, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 42 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:10.655489 28456 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:10.655589 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 42 FOLLOWER]: Advancing to term 43
I20250113 21:10:10.656252 28456 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.656405 28456 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 43 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:10.656618 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 43 candidate_status { last_received { term: 41 index: 65 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:10.656618 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 43 candidate_status { last_received { term: 41 index: 65 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:10.656720 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 42 FOLLOWER]: Advancing to term 43
I20250113 21:10:10.657276 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 43.
I20250113 21:10:10.657438 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:10.657528 28456 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 43 FOLLOWER]: Leader election won for term 43
I20250113 21:10:10.657645 28456 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 43 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:10.657706 28456 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 41.65, Last appended by leader: 65, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.658170 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 42 to 43, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 43 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:10.701488 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 43 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 42. Current term is 43. Ops: []
I20250113 21:10:10.701742 28451 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250113 21:10:10.701850 28451 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 43.66 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 65
I20250113 21:10:10.701915 28461 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 42 LEADER]: Stepping down as leader of term 42
I20250113 21:10:10.701964 28461 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:10.702023 28461 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 42.66, Last appended by leader: 66, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.702100 28461 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 42 FOLLOWER]: Advancing to term 43
I20250113 21:10:10.726106 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:10.741413 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 43 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 41 index: 65. Preceding OpId from leader: term: 43 index: 66. (index mismatch)
I20250113 21:10:10.741676 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250113 21:10:10.742740 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 43 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 42 index: 66. Preceding OpId from leader: term: 43 index: 66. (term mismatch)
I20250113 21:10:10.742794 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 65
I20250113 21:10:10.742816 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 42.66
I20250113 21:10:10.742856 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:10.742993 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250113 21:10:10.843161 28460 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:10.843251 28460 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 43 FOLLOWER]: Advancing to term 44
I20250113 21:10:10.843919 28460 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.844074 28460 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 44 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:10.844285 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 44 candidate_status { last_received { term: 43 index: 66 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:10.844285 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 44 candidate_status { last_received { term: 43 index: 66 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:10.844537 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:10.844624 28460 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20250113 21:10:10.862876 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 44 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 43. Current term is 44. Ops: []
I20250113 21:10:10.863106 28456 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 43.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250113 21:10:10.863235 28456 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 43 LEADER]: Stepping down as leader of term 43
I20250113 21:10:10.863277 28456 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:10.863330 28456 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 43.66, Last appended by leader: 66, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.863402 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 43 FOLLOWER]: Advancing to term 44
I20250113 21:10:10.927599 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 43 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:10.927703 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 43 FOLLOWER]: Advancing to term 44
I20250113 21:10:10.928367 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.928529 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 44 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:10.928755 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 44 candidate_status { last_received { term: 43 index: 66 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:10.928755 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 44 candidate_status { last_received { term: 43 index: 66 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:10.928876 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 44: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:10.929374 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 44.
I20250113 21:10:10.929567 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:10.929651 28451 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 44 FOLLOWER]: Leader election won for term 44
I20250113 21:10:10.929702 28451 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 44 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:10.929742 28451 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 43.66, Last appended by leader: 66, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.930244 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 43 to 44, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 44 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:10.971377 28460 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:10.971478 28460 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 44 FOLLOWER]: Advancing to term 45
I20250113 21:10:10.972136 28460 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.972301 28460 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 45 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:10.972524 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 45 candidate_status { last_received { term: 43 index: 66 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:10.972524 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 45 candidate_status { last_received { term: 43 index: 66 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:10.972652 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 44 FOLLOWER]: Advancing to term 45
I20250113 21:10:10.973199 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 45.
I20250113 21:10:10.973352 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:10.973469 28460 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 45 FOLLOWER]: Leader election won for term 45
I20250113 21:10:10.973532 28460 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 45 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:10.973585 28460 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 43.66, Last appended by leader: 66, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:10.974035 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 44 to 45, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 45 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:11.008702 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 45 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 44. Current term is 45. Ops: []
I20250113 21:10:11.008965 28451 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250113 21:10:11.009088 28451 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 44 LEADER]: Stepping down as leader of term 44
I20250113 21:10:11.009122 28451 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:11.009207 28451 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 44.67, Last appended by leader: 67, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.009269 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 44 FOLLOWER]: Advancing to term 45
I20250113 21:10:11.073663 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 45 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 43 index: 66. Preceding OpId from leader: term: 45 index: 67. (index mismatch)
I20250113 21:10:11.073953 28460 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250113 21:10:11.074913 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 45 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 44 index: 67. Preceding OpId from leader: term: 45 index: 67. (term mismatch)
I20250113 21:10:11.074970 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 66
I20250113 21:10:11.075003 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 44.67
I20250113 21:10:11.075026 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:11.075232 28460 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250113 21:10:11.257913 28461 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 45 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:11.258018 28461 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 45 FOLLOWER]: Advancing to term 46
I20250113 21:10:11.258720 28461 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.258949 28461 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 46 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:11.259011 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 46 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 45. Current term is 46. Ops: []
I20250113 21:10:11.259017 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 46 candidate_status { last_received { term: 45 index: 67 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:11.259093 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 46 candidate_status { last_received { term: 45 index: 67 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:11.259349 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:11.259424 28461 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20250113 21:10:11.259392 28463 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 45.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250113 21:10:11.259529 28455 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 45 LEADER]: Stepping down as leader of term 45
I20250113 21:10:11.259572 28455 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:11.259629 28455 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 45.67, Last appended by leader: 67, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.259701 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 45 FOLLOWER]: Advancing to term 46
I20250113 21:10:11.340638 28456 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 45 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:11.340726 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 45 FOLLOWER]: Advancing to term 46
I20250113 21:10:11.341403 28456 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.341574 28456 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 46 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:11.341769 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 46 candidate_status { last_received { term: 45 index: 67 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:11.341840 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 46: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:11.341780 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 46 candidate_status { last_received { term: 45 index: 67 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:11.342516 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 46.
I20250113 21:10:11.342661 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:11.342732 28456 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 46 FOLLOWER]: Leader election won for term 46
I20250113 21:10:11.342792 28456 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 46 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:11.342844 28456 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 45.67, Last appended by leader: 67, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.343233 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 45 to 46, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 46 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:11.376749 28461 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:11.376868 28461 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 46 FOLLOWER]: Advancing to term 47
I20250113 21:10:11.377617 28461 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.377785 28461 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 47 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:11.378029 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 47 candidate_status { last_received { term: 45 index: 67 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:11.378036 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 47 candidate_status { last_received { term: 45 index: 67 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:11.378139 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 46 FOLLOWER]: Advancing to term 47
I20250113 21:10:11.378723 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 47.
I20250113 21:10:11.378885 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:11.378984 28461 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 47 FOLLOWER]: Leader election won for term 47
I20250113 21:10:11.379040 28461 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 47 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:11.379093 28461 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 45.67, Last appended by leader: 67, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.379524 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 46 to 47, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 47 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:11.447824 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 47 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 46. Current term is 47. Ops: []
I20250113 21:10:11.448098 28456 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250113 21:10:11.448184 28456 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 47.68 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 67
I20250113 21:10:11.448206 28456 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 46 LEADER]: Stepping down as leader of term 46
I20250113 21:10:11.448236 28456 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:11.448287 28456 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 46.68, Last appended by leader: 68, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.448361 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 46 FOLLOWER]: Advancing to term 47
I20250113 21:10:11.469740 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 47 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 46 index: 68. Preceding OpId from leader: term: 47 index: 68. (term mismatch)
I20250113 21:10:11.469823 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 67
I20250113 21:10:11.469856 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 46.68
I20250113 21:10:11.469924 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:11.470134 28461 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250113 21:10:11.471105 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 47 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 45 index: 67. Preceding OpId from leader: term: 47 index: 68. (index mismatch)
I20250113 21:10:11.471297 28461 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250113 21:10:11.663619 28455 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:11.663769 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 47 FOLLOWER]: Advancing to term 48
I20250113 21:10:11.664497 28455 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.664674 28455 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 48 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:11.664897 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 48 candidate_status { last_received { term: 47 index: 68 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:11.664897 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 48 candidate_status { last_received { term: 47 index: 68 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:11.665154 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:11.665251 28455 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20250113 21:10:11.671484 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 48 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 47. Current term is 48. Ops: []
I20250113 21:10:11.671700 28464 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 47.68, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250113 21:10:11.671811 28464 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 47 LEADER]: Stepping down as leader of term 47
I20250113 21:10:11.671838 28464 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:11.671900 28464 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 47.68, Last appended by leader: 68, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.671972 28464 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 47 FOLLOWER]: Advancing to term 48
I20250113 21:10:11.687530 28456 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 47 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:11.687598 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 47 FOLLOWER]: Advancing to term 48
I20250113 21:10:11.688158 28456 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.688311 28456 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 48 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:11.688496 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 48 candidate_status { last_received { term: 47 index: 68 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:11.688508 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 48 candidate_status { last_received { term: 47 index: 68 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:11.688594 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 48: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:11.689095 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 48.
I20250113 21:10:11.689263 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:11.689353 28456 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 48 FOLLOWER]: Leader election won for term 48
I20250113 21:10:11.689428 28456 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 48 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:11.689484 28456 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 47.68, Last appended by leader: 68, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.689981 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 47 to 48, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 48 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:11.726408 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:11.774417 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 48 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 47 index: 68. Preceding OpId from leader: term: 48 index: 69. (index mismatch)
I20250113 21:10:11.774672 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250113 21:10:11.775532 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 48 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 47 index: 68. Preceding OpId from leader: term: 48 index: 69. (index mismatch)
I20250113 21:10:11.775810 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250113 21:10:11.876318 28455 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:11.876417 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 48 FOLLOWER]: Advancing to term 49
I20250113 21:10:11.877071 28455 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.877229 28455 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 49 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:11.877461 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 49 candidate_status { last_received { term: 48 index: 69 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:11.877431 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 49 candidate_status { last_received { term: 48 index: 69 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:11.877717 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:11.877806 28455 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20250113 21:10:11.888077 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 49 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 48. Current term is 49. Ops: []
I20250113 21:10:11.888321 28462 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 48.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250113 21:10:11.888445 28462 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 48 LEADER]: Stepping down as leader of term 48
I20250113 21:10:11.888478 28462 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:11.888530 28462 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 48.69, Last appended by leader: 69, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.888607 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 48 FOLLOWER]: Advancing to term 49
I20250113 21:10:11.963483 28464 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 48 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:11.963595 28464 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 48 FOLLOWER]: Advancing to term 49
I20250113 21:10:11.964336 28464 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.964490 28464 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 49 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:11.964730 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 49 candidate_status { last_received { term: 48 index: 69 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:11.964807 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 49: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:11.964737 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 49 candidate_status { last_received { term: 48 index: 69 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:11.965394 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 49.
I20250113 21:10:11.965571 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:11.965662 28464 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 49 FOLLOWER]: Leader election won for term 49
I20250113 21:10:11.965719 28464 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 49 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:11.965792 28464 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 48.69, Last appended by leader: 69, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:11.966275 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 48 to 49, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 49 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:12.004173 28455 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:12.004283 28455 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 49 FOLLOWER]: Advancing to term 50
I20250113 21:10:12.004973 28455 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.005154 28455 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 50 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:12.005354 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 50 candidate_status { last_received { term: 48 index: 69 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:12.005363 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 50 candidate_status { last_received { term: 48 index: 69 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:12.005468 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 49 FOLLOWER]: Advancing to term 50
I20250113 21:10:12.006104 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 50.
I20250113 21:10:12.006265 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:12.006369 28455 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 50 FOLLOWER]: Leader election won for term 50
I20250113 21:10:12.006424 28455 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 50 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:12.006476 28455 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 48.69, Last appended by leader: 69, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.006986 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 49 to 50, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 50 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:12.080765 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 50 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 49. Current term is 50. Ops: []
I20250113 21:10:12.081009 28464 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250113 21:10:12.081122 28464 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 50.70 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69
I20250113 21:10:12.081166 28451 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 49 LEADER]: Stepping down as leader of term 49
I20250113 21:10:12.081203 28451 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:12.081252 28451 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 49.70, Last appended by leader: 70, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.081324 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 49 FOLLOWER]: Advancing to term 50
I20250113 21:10:12.085734 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 50 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 49 index: 70. Preceding OpId from leader: term: 50 index: 70. (term mismatch)
I20250113 21:10:12.085788 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 69
I20250113 21:10:12.085809 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 49.70
I20250113 21:10:12.085830 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:12.086063 28455 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250113 21:10:12.087018 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 50 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 48 index: 69. Preceding OpId from leader: term: 50 index: 70. (index mismatch)
I20250113 21:10:12.087198 28465 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250113 21:10:12.188279 28456 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 50 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:12.188382 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 50 FOLLOWER]: Advancing to term 51
I20250113 21:10:12.189046 28456 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.189201 28456 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 51 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:12.189419 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 51 candidate_status { last_received { term: 50 index: 70 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:12.189450 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 51 candidate_status { last_received { term: 50 index: 70 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:12.189553 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 50 FOLLOWER]: Advancing to term 51
I20250113 21:10:12.189610 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 51 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 50. Current term is 51. Ops: []
I20250113 21:10:12.190085 28465 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 50.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250113 21:10:12.190237 28466 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 50 LEADER]: Stepping down as leader of term 50
I20250113 21:10:12.190287 28466 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:12.190351 28466 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 50.70, Last appended by leader: 70, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.190361 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 51.
I20250113 21:10:12.190448 28466 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 50 FOLLOWER]: Advancing to term 51
I20250113 21:10:12.190559 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:12.190658 28456 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 51 FOLLOWER]: Leader election won for term 51
I20250113 21:10:12.190712 28456 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 51 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:12.190755 28456 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 50.70, Last appended by leader: 70, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.191113 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 50 to 51, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 51 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:12.298537 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 51 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 50 index: 70. Preceding OpId from leader: term: 51 index: 71. (index mismatch)
I20250113 21:10:12.298825 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250113 21:10:12.299383 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 51 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 50 index: 70. Preceding OpId from leader: term: 51 index: 71. (index mismatch)
I20250113 21:10:12.299579 28456 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250113 21:10:12.486292 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 51 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:12.486395 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 51 FOLLOWER]: Advancing to term 52
I20250113 21:10:12.487051 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.487219 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 52 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:12.487478 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 52 candidate_status { last_received { term: 51 index: 71 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:12.487478 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 52 candidate_status { last_received { term: 51 index: 71 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:12.487774 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:12.487883 28451 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20250113 21:10:12.489059 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 52 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 51. Current term is 52. Ops: []
I20250113 21:10:12.489251 28462 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 51.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250113 21:10:12.489329 28462 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 51 LEADER]: Stepping down as leader of term 51
I20250113 21:10:12.489359 28462 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:12.489435 28462 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 51.71, Last appended by leader: 71, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.489531 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 51 FOLLOWER]: Advancing to term 52
I20250113 21:10:12.496872 28466 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 51 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:12.496928 28466 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 51 FOLLOWER]: Advancing to term 52
I20250113 21:10:12.497445 28466 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.497591 28466 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 52 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:12.497776 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 52 candidate_status { last_received { term: 51 index: 71 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:12.497886 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 52: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:12.497838 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 52 candidate_status { last_received { term: 51 index: 71 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:12.498399 28180 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 52.
I20250113 21:10:12.498561 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:12.498644 28466 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 52 FOLLOWER]: Leader election won for term 52
I20250113 21:10:12.498696 28466 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 52 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:12.498852 28466 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 51.71, Last appended by leader: 71, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.499223 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 51 to 52, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 52 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:12.598683 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 52 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 51 index: 71. Preceding OpId from leader: term: 52 index: 72. (index mismatch)
I20250113 21:10:12.598973 28466 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250113 21:10:12.600078 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 52 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 51 index: 71. Preceding OpId from leader: term: 52 index: 72. (index mismatch)
I20250113 21:10:12.600260 28465 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250113 21:10:12.701581 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 52 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:12.701737 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 52 FOLLOWER]: Advancing to term 53
I20250113 21:10:12.702553 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.702687 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 53 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 52. Current term is 53. Ops: []
I20250113 21:10:12.702804 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 53 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:12.702864 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 53 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:12.702880 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 53 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:12.703002 28466 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 52.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250113 21:10:12.703078 28466 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 52 LEADER]: Stepping down as leader of term 52
I20250113 21:10:12.703109 28466 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:12.703137 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:12.703181 28466 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 52.72, Last appended by leader: 72, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.703262 28466 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 52 FOLLOWER]: Advancing to term 53
I20250113 21:10:12.703233 28451 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20250113 21:10:12.726684 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:12.794221 28462 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 52 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:12.794329 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 52 FOLLOWER]: Advancing to term 53
I20250113 21:10:12.795035 28462 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.795208 28462 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 53 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:12.795429 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 53 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:12.795434 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 53 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:12.795553 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 53: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:12.796069 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 53.
I20250113 21:10:12.796248 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:12.796336 28462 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 53 FOLLOWER]: Leader election won for term 53
I20250113 21:10:12.796404 28462 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 53 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:12.796458 28462 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 52.72, Last appended by leader: 72, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.796902 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 52 to 53, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 53 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:12.828666 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:12.828774 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 53 FOLLOWER]: Advancing to term 54
I20250113 21:10:12.829388 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.829550 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 54 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:12.829777 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 54 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:12.829777 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 54 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:12.829924 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 53 FOLLOWER]: Advancing to term 54
I20250113 21:10:12.830466 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 54.
I20250113 21:10:12.830659 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:12.830744 28451 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 54 FOLLOWER]: Leader election won for term 54
I20250113 21:10:12.830849 28451 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 54 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:12.830896 28451 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 52.72, Last appended by leader: 72, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.831343 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 53 to 54, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 54 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:12.878542 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 54 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 53. Current term is 54. Ops: []
I20250113 21:10:12.878829 28462 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250113 21:10:12.878932 28462 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 53 LEADER]: Stepping down as leader of term 53
I20250113 21:10:12.878968 28462 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:12.879021 28462 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 53.73, Last appended by leader: 73, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.879092 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 53 FOLLOWER]: Advancing to term 54
I20250113 21:10:12.931041 28466 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:12.931139 28466 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 54 FOLLOWER]: Advancing to term 55
I20250113 21:10:12.931808 28466 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.931977 28466 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 55 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:12.932179 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 55 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:12.932180 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 55 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:12.932296 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 54 FOLLOWER]: Advancing to term 55
I20250113 21:10:12.932876 28180 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 55 because replica has last-logged OpId of term: 53 index: 73, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 72.
I20250113 21:10:12.933039 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:12.933125 28466 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20250113 21:10:12.936723 28180 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 55 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 54. Current term is 55. Ops: []
I20250113 21:10:12.936959 28451 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250113 21:10:12.937033 28451 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 53.73 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 72
I20250113 21:10:12.937076 28468 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 54 LEADER]: Stepping down as leader of term 54
I20250113 21:10:12.937115 28468 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:12.937172 28468 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 54.73, Last appended by leader: 73, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:12.937247 28468 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 54 FOLLOWER]: Advancing to term 55
I20250113 21:10:13.072571 28466 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:13.072670 28466 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 55 FOLLOWER]: Advancing to term 56
I20250113 21:10:13.073313 28466 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.073472 28466 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 56 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:13.073704 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 56 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:13.073769 28180 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 55 FOLLOWER]: Advancing to term 56
I20250113 21:10:13.073707 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 56 candidate_status { last_received { term: 52 index: 72 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:13.073828 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 55 FOLLOWER]: Advancing to term 56
I20250113 21:10:13.074345 28180 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 56 because replica has last-logged OpId of term: 53 index: 73, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 72.
I20250113 21:10:13.074414 28256 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 56 because replica has last-logged OpId of term: 54 index: 73, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 72.
I20250113 21:10:13.074700 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:13.074836 28466 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20250113 21:10:13.192852 28462 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:13.192960 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 56 FOLLOWER]: Advancing to term 57
I20250113 21:10:13.193604 28462 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.193747 28462 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 57 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:13.193979 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 57 candidate_status { last_received { term: 53 index: 73 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:13.194051 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 56 FOLLOWER]: Advancing to term 57
I20250113 21:10:13.193982 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 57 candidate_status { last_received { term: 53 index: 73 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:13.194089 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 56 FOLLOWER]: Advancing to term 57
I20250113 21:10:13.194545 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 57.
I20250113 21:10:13.194736 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters:
I20250113 21:10:13.194809 28256 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b for term 57 because replica has last-logged OpId of term: 54 index: 73, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 73.
I20250113 21:10:13.194823 28462 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 57 FOLLOWER]: Leader election won for term 57
I20250113 21:10:13.194886 28462 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 57 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:13.194939 28462 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 53.73, Last appended by leader: 73, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.195358 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 54 to 57, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 57 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:13.259302 28468 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:13.259409 28468 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 57 FOLLOWER]: Advancing to term 58
I20250113 21:10:13.260183 28468 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.260353 28468 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 58 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:13.260553 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 58 candidate_status { last_received { term: 54 index: 73 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:13.260574 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 58 candidate_status { last_received { term: 54 index: 73 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:13.260651 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 57 FOLLOWER]: Advancing to term 58
I20250113 21:10:13.261265 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 58.
I20250113 21:10:13.261435 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:13.261540 28468 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 58 FOLLOWER]: Leader election won for term 58
I20250113 21:10:13.261587 28468 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 58 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:13.261632 28468 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 54.73, Last appended by leader: 73, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.262151 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 57 to 58, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 58 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:13.286784 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 58 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 57. Current term is 58. Ops: []
I20250113 21:10:13.287050 28462 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20250113 21:10:13.287143 28462 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 58.74 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 72
I20250113 21:10:13.287178 28456 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 57 LEADER]: Stepping down as leader of term 57
I20250113 21:10:13.287235 28456 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 57 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:13.287302 28456 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 57.74, Last appended by leader: 74, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.287381 28456 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 57 FOLLOWER]: Advancing to term 58
I20250113 21:10:13.356715 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 58 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 52 index: 72. Preceding OpId from leader: term: 58 index: 74. (index mismatch)
I20250113 21:10:13.356979 28468 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20250113 21:10:13.358079 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 58 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 57 index: 74. Preceding OpId from leader: term: 58 index: 74. (term mismatch)
I20250113 21:10:13.358129 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 73
I20250113 21:10:13.358147 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 57.74
I20250113 21:10:13.358170 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:13.358326 28468 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20250113 21:10:13.358456 28468 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 53.73 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 72
I20250113 21:10:13.358673 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 72
I20250113 21:10:13.358709 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 53.73
I20250113 21:10:13.358733 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:13.459087 28469 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 58 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:13.459193 28469 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 58 FOLLOWER]: Advancing to term 59
I20250113 21:10:13.459959 28469 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.460110 28469 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 59 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:13.460316 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 59 candidate_status { last_received { term: 58 index: 74 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:13.460341 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 59 candidate_status { last_received { term: 58 index: 74 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:13.460537 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 59 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:13.460630 28469 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 59 FOLLOWER]: Leader election lost for term 59. Reason: could not achieve majority
I20250113 21:10:13.472268 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 59 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 58. Current term is 59. Ops: []
I20250113 21:10:13.472497 28470 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 58.74, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250113 21:10:13.472600 28470 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 58 LEADER]: Stepping down as leader of term 58
I20250113 21:10:13.472623 28470 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:13.472688 28470 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 58.74, Last appended by leader: 74, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.472762 28470 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 58 FOLLOWER]: Advancing to term 59
I20250113 21:10:13.540602 28462 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 58 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:13.540683 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 58 FOLLOWER]: Advancing to term 59
I20250113 21:10:13.541309 28462 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.541455 28462 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 59 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:13.541674 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 59 candidate_status { last_received { term: 58 index: 74 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:13.541679 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 59 candidate_status { last_received { term: 58 index: 74 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:13.541791 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 59: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:13.542313 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 59.
I20250113 21:10:13.542476 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:13.542598 28462 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 59 FOLLOWER]: Leader election won for term 59
I20250113 21:10:13.542655 28462 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 59 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:13.542706 28462 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 58.74, Last appended by leader: 74, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.543123 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 58 to 59, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 59 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:13.566928 28469 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 59 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:13.567001 28469 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 59 FOLLOWER]: Advancing to term 60
I20250113 21:10:13.567539 28469 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.567664 28469 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 60 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:13.567853 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 60 candidate_status { last_received { term: 58 index: 74 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:13.567838 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 60 candidate_status { last_received { term: 58 index: 74 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:13.567942 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 59 FOLLOWER]: Advancing to term 60
I20250113 21:10:13.568537 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 60.
I20250113 21:10:13.568687 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:13.568758 28469 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 60 FOLLOWER]: Leader election won for term 60
I20250113 21:10:13.568819 28469 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 60 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:13.568882 28469 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 58.74, Last appended by leader: 74, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.569321 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 59 to 60, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 60 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:13.632920 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 60 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 59. Current term is 60. Ops: []
I20250113 21:10:13.633184 28462 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250113 21:10:13.633301 28462 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 59 LEADER]: Stepping down as leader of term 59
I20250113 21:10:13.633337 28462 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:13.633404 28462 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 59.75, Last appended by leader: 75, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.633474 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 59 FOLLOWER]: Advancing to term 60
I20250113 21:10:13.657649 28180 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 60 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 59 index: 75. Preceding OpId from leader: term: 60 index: 75. (term mismatch)
I20250113 21:10:13.657723 28180 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 74
I20250113 21:10:13.657747 28180 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 59.75
I20250113 21:10:13.657769 28180 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 60 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:13.658052 28469 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250113 21:10:13.658979 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 60 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 58 index: 74. Preceding OpId from leader: term: 60 index: 75. (index mismatch)
I20250113 21:10:13.659237 28469 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250113 21:10:13.726938 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:13.839699 28451 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 60 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:13.839803 28451 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 60 FOLLOWER]: Advancing to term 61
I20250113 21:10:13.840463 28451 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.840636 28451 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 61 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:13.840854 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 61 candidate_status { last_received { term: 60 index: 75 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:13.840867 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 61 candidate_status { last_received { term: 60 index: 75 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:13.841101 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:13.841182 28451 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20250113 21:10:13.845139 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 61 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 60. Current term is 61. Ops: []
I20250113 21:10:13.845326 28471 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 60.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250113 21:10:13.845410 28471 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 60 LEADER]: Stepping down as leader of term 60
I20250113 21:10:13.845444 28471 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 60 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:13.845497 28471 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 60.75, Last appended by leader: 75, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.845571 28471 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 60 FOLLOWER]: Advancing to term 61
I20250113 21:10:13.859290 28462 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 60 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:13.859369 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 60 FOLLOWER]: Advancing to term 61
I20250113 21:10:13.859992 28462 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.860154 28462 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 61 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:13.860337 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 61 candidate_status { last_received { term: 60 index: 75 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:13.860342 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 61 candidate_status { last_received { term: 60 index: 75 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:13.860448 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 61: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:13.861002 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 61.
I20250113 21:10:13.861164 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:13.861250 28462 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 61 FOLLOWER]: Leader election won for term 61
I20250113 21:10:13.861299 28462 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 61 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:13.861348 28462 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 60.75, Last appended by leader: 75, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:13.861792 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 60 to 61, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 61 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:13.976365 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 61 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 60 index: 75. Preceding OpId from leader: term: 61 index: 76. (index mismatch)
I20250113 21:10:13.976648 28462 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250113 21:10:13.977650 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 61 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 60 index: 75. Preceding OpId from leader: term: 61 index: 76. (index mismatch)
I20250113 21:10:13.977896 28462 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250113 21:10:14.078320 28471 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 61 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:14.078421 28471 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 61 FOLLOWER]: Advancing to term 62
I20250113 21:10:14.078706 28470 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 61 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:14.078774 28470 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 61 FOLLOWER]: Advancing to term 62
I20250113 21:10:14.079173 28471 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.079330 28471 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 62 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.079355 28470 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.079516 28180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 62 candidate_status { last_received { term: 61 index: 76 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.079564 28470 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 62 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.079625 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 62 candidate_status { last_received { term: 61 index: 76 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:14.079713 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 62 candidate_status { last_received { term: 61 index: 76 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.079716 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 62: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:14.079524 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 62 candidate_status { last_received { term: 61 index: 76 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:14.079885 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 62: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:14.080036 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:14.080046 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:14.080142 28470 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20250113 21:10:14.080147 28471 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20250113 21:10:14.097693 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 62 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 61. Current term is 62. Ops: []
I20250113 21:10:14.097988 28462 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 61.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250113 21:10:14.098104 28462 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 61 LEADER]: Stepping down as leader of term 61
I20250113 21:10:14.098130 28462 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 61 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:14.098194 28462 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 61.76, Last appended by leader: 76, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.098276 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 61 FOLLOWER]: Advancing to term 62
I20250113 21:10:14.180547 28471 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 62 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:14.180649 28471 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 62 FOLLOWER]: Advancing to term 63
I20250113 21:10:14.181321 28471 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.181526 28471 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 63 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.181757 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 63 candidate_status { last_received { term: 61 index: 76 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.181824 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 62 FOLLOWER]: Advancing to term 63
I20250113 21:10:14.181763 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 63 candidate_status { last_received { term: 61 index: 76 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:14.181890 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 62 FOLLOWER]: Advancing to term 63
I20250113 21:10:14.182439 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 63.
I20250113 21:10:14.182461 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 63.
I20250113 21:10:14.182628 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters:
I20250113 21:10:14.182709 28471 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 63 FOLLOWER]: Leader election won for term 63
I20250113 21:10:14.182785 28471 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 63 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:14.182833 28471 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 61.76, Last appended by leader: 76, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.183259 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 61 to 63, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 63 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:14.294817 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 63 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 61 index: 76. Preceding OpId from leader: term: 63 index: 77. (index mismatch)
I20250113 21:10:14.295091 28471 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250113 21:10:14.296048 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 63 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 61 index: 76. Preceding OpId from leader: term: 63 index: 77. (index mismatch)
I20250113 21:10:14.296236 28471 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250113 21:10:14.397043 28470 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 63 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:14.397145 28470 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 63 FOLLOWER]: Advancing to term 64
I20250113 21:10:14.397856 28470 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.398051 28470 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 64 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.398236 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 64 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:14.398253 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 64 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.398461 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:14.398545 28470 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I20250113 21:10:14.405903 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 64 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 63. Current term is 64. Ops: []
I20250113 21:10:14.406111 28473 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 63.77, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250113 21:10:14.406263 28469 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 63 LEADER]: Stepping down as leader of term 63
I20250113 21:10:14.406311 28469 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 63 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:14.406374 28469 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 63.77, Last appended by leader: 77, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.406452 28469 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 63 FOLLOWER]: Advancing to term 64
I20250113 21:10:14.483219 28462 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 63 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:14.483318 28462 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 63 FOLLOWER]: Advancing to term 64
I20250113 21:10:14.484905 28462 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.485074 28462 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 64 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:14.485286 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 64 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:14.485311 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 64 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:14.485396 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 64 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 64: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:14.485930 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 64.
I20250113 21:10:14.486114 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:14.486204 28462 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 64 FOLLOWER]: Leader election won for term 64
I20250113 21:10:14.486269 28462 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 64 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:14.486322 28462 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 63.77, Last appended by leader: 77, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.486819 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 63 to 64, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 64 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:14.547084 28470 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:14.547181 28470 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 64 FOLLOWER]: Advancing to term 65
I20250113 21:10:14.547878 28470 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.548045 28470 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 65 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.548249 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 65 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:14.548319 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 64 FOLLOWER]: Advancing to term 65
I20250113 21:10:14.548262 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 65 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.548925 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 65.
I20250113 21:10:14.549091 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:14.549178 28470 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 65 FOLLOWER]: Leader election won for term 65
I20250113 21:10:14.549292 28470 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 65 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:14.549337 28470 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 63.77, Last appended by leader: 77, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.549759 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 64 to 65, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 65 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:14.579036 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 65 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 64. Current term is 65. Ops: []
I20250113 21:10:14.579335 28462 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250113 21:10:14.579445 28462 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 65.78 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 77
I20250113 21:10:14.579488 28472 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 64 LEADER]: Stepping down as leader of term 64
I20250113 21:10:14.579538 28472 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 64 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:14.579613 28472 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 64.78, Last appended by leader: 78, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.579700 28472 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 64 FOLLOWER]: Advancing to term 65
I20250113 21:10:14.658470 28469 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 65 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:14.658587 28469 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 65 FOLLOWER]: Advancing to term 66
I20250113 21:10:14.659271 28469 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.659447 28469 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 66 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.659660 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 66 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.659660 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 66 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:14.659759 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 65 FOLLOWER]: Advancing to term 66
I20250113 21:10:14.660336 28179 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 66 because replica has last-logged OpId of term: 64 index: 78, which is greater than that of the candidate, which has last-logged OpId of term: 63 index: 77.
I20250113 21:10:14.660461 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:14.660544 28469 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20250113 21:10:14.673251 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 66 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 65. Current term is 66. Ops: []
I20250113 21:10:14.673255 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 66 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 65. Current term is 66. Ops: []
I20250113 21:10:14.673498 28470 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250113 21:10:14.673681 28470 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 65 LEADER]: Stepping down as leader of term 65
I20250113 21:10:14.673727 28470 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 65 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:14.673693 28474 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250113 21:10:14.673843 28474 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 64.78 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 77
I20250113 21:10:14.673940 28470 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 65.78, Last appended by leader: 78, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.674018 28470 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 65 FOLLOWER]: Advancing to term 66
I20250113 21:10:14.727195 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:14.763840 28469 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:14.763944 28469 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 66 FOLLOWER]: Advancing to term 67
I20250113 21:10:14.764597 28469 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.764762 28469 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 67 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.764956 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 67 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.764977 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 67 candidate_status { last_received { term: 63 index: 77 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:14.765074 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 66 FOLLOWER]: Advancing to term 67
I20250113 21:10:14.765081 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 66 FOLLOWER]: Advancing to term 67
I20250113 21:10:14.765772 28256 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 67 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 67 because replica has last-logged OpId of term: 65 index: 78, which is greater than that of the candidate, which has last-logged OpId of term: 63 index: 77.
I20250113 21:10:14.765772 28179 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 67 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 67 because replica has last-logged OpId of term: 64 index: 78, which is greater than that of the candidate, which has last-logged OpId of term: 63 index: 77.
I20250113 21:10:14.766073 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 67 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:14.766163 28469 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 67 FOLLOWER]: Leader election lost for term 67. Reason: could not achieve majority
I20250113 21:10:14.815338 28472 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 67 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:14.815443 28472 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 67 FOLLOWER]: Advancing to term 68
I20250113 21:10:14.816141 28472 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.816308 28472 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 68 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:14.816510 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 68 candidate_status { last_received { term: 64 index: 78 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:14.816576 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 67 FOLLOWER]: Advancing to term 68
I20250113 21:10:14.816538 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 68 candidate_status { last_received { term: 64 index: 78 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:14.816618 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 67 FOLLOWER]: Advancing to term 68
I20250113 21:10:14.817111 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 68.
I20250113 21:10:14.817210 28256 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 68 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b for term 68 because replica has last-logged OpId of term: 65 index: 78, which is greater than that of the candidate, which has last-logged OpId of term: 64 index: 78.
I20250113 21:10:14.817305 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 68 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters:
I20250113 21:10:14.817395 28472 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 68 FOLLOWER]: Leader election won for term 68
I20250113 21:10:14.817442 28472 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 68 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:14.817507 28472 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 64.78, Last appended by leader: 78, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.817967 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 65 to 68, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 68 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:14.896234 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 68 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 65 index: 78. Preceding OpId from leader: term: 68 index: 79. (index mismatch)
I20250113 21:10:14.896482 28472 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20250113 21:10:14.896569 28472 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 65.78 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 77
I20250113 21:10:14.896768 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 77
I20250113 21:10:14.896806 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 65.78
I20250113 21:10:14.896831 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 68 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:14.896905 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 68 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 63 index: 77. Preceding OpId from leader: term: 68 index: 79. (index mismatch)
I20250113 21:10:14.897123 28472 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20250113 21:10:14.998504 28473 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 68 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:14.998589 28473 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 68 FOLLOWER]: Advancing to term 69
I20250113 21:10:14.999315 28473 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:14.999462 28473 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 69 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:14.999612 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 69 candidate_status { last_received { term: 68 index: 79 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:14.999624 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 69 candidate_status { last_received { term: 68 index: 79 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:14.999841 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 69 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:14.999926 28473 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 69 FOLLOWER]: Leader election lost for term 69. Reason: could not achieve majority
I20250113 21:10:15.008716 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 69 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 68. Current term is 69. Ops: []
I20250113 21:10:15.008931 28472 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 68.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250113 21:10:15.009033 28472 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 68 LEADER]: Stepping down as leader of term 68
I20250113 21:10:15.009065 28472 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 68 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:15.009128 28472 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 68.79, Last appended by leader: 79, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.009202 28472 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 68 FOLLOWER]: Advancing to term 69
I20250113 21:10:15.097633 28474 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 68 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:15.097739 28474 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 68 FOLLOWER]: Advancing to term 69
I20250113 21:10:15.098533 28474 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.098702 28474 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 69 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:15.098912 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 69 candidate_status { last_received { term: 68 index: 79 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:15.098912 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 69 candidate_status { last_received { term: 68 index: 79 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:15.099010 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 69: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:15.099515 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 69.
I20250113 21:10:15.099676 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:15.099782 28474 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 69 FOLLOWER]: Leader election won for term 69
I20250113 21:10:15.099843 28474 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 69 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:15.099903 28474 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 68.79, Last appended by leader: 79, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.100378 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 68 to 69, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 69 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:15.129315 28473 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:15.129423 28473 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 69 FOLLOWER]: Advancing to term 70
I20250113 21:10:15.130174 28473 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.130371 28473 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 70 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:15.130563 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 70 candidate_status { last_received { term: 68 index: 79 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:15.130628 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 69 FOLLOWER]: Advancing to term 70
I20250113 21:10:15.130573 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 70 candidate_status { last_received { term: 68 index: 79 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:15.131160 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 70.
I20250113 21:10:15.131297 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:15.131384 28473 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 70 FOLLOWER]: Leader election won for term 70
I20250113 21:10:15.131433 28473 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 70 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:15.131497 28473 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 68.79, Last appended by leader: 79, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.131909 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 69 to 70, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 70 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:15.176370 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 70 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 69. Current term is 70. Ops: []
I20250113 21:10:15.176628 28474 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250113 21:10:15.176725 28474 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 69 LEADER]: Stepping down as leader of term 69
I20250113 21:10:15.176756 28474 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 69 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:15.176815 28474 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 69.80, Last appended by leader: 80, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.176877 28474 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 69 FOLLOWER]: Advancing to term 70
I20250113 21:10:15.229473 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 70 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 69 index: 80. Preceding OpId from leader: term: 70 index: 80. (term mismatch)
I20250113 21:10:15.229553 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 79
I20250113 21:10:15.229583 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 69.80
I20250113 21:10:15.229609 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 70 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:15.229880 28473 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250113 21:10:15.230759 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 70 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 68 index: 79. Preceding OpId from leader: term: 70 index: 80. (index mismatch)
I20250113 21:10:15.230948 28476 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250113 21:10:15.331794 28472 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 70 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:15.331899 28472 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 70 FOLLOWER]: Advancing to term 71
I20250113 21:10:15.332549 28472 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.332719 28472 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 71 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:15.332917 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 71 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:15.332919 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 71 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:15.333070 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 70 FOLLOWER]: Advancing to term 71
I20250113 21:10:15.333626 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 71.
I20250113 21:10:15.333781 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 71 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:15.333881 28472 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 71 FOLLOWER]: Leader election won for term 71
I20250113 21:10:15.333940 28472 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 71 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:15.333986 28472 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 70.80, Last appended by leader: 80, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.334404 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 70 to 71, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 71 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:15.348783 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 71 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 70. Current term is 71. Ops: []
I20250113 21:10:15.349031 28473 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 70.80, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250113 21:10:15.349123 28473 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 71.81 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 80
I20250113 21:10:15.349190 28476 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 70 LEADER]: Stepping down as leader of term 70
I20250113 21:10:15.349248 28476 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 70 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:15.349303 28476 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 70.80, Last appended by leader: 80, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.349395 28476 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 70 FOLLOWER]: Advancing to term 71
I20250113 21:10:15.410816 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 71 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 70 index: 80. Preceding OpId from leader: term: 71 index: 81. (index mismatch)
I20250113 21:10:15.411087 28472 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250113 21:10:15.412012 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 71 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 70 index: 80. Preceding OpId from leader: term: 71 index: 81. (index mismatch)
I20250113 21:10:15.412197 28472 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250113 21:10:15.692184 28474 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 71 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:15.692301 28474 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 71 FOLLOWER]: Advancing to term 72
I20250113 21:10:15.693295 28474 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.693535 28474 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 72 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:15.693773 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 72 candidate_status { last_received { term: 71 index: 81 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:15.693773 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 72 candidate_status { last_received { term: 71 index: 81 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:15.694114 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 72 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:15.694213 28474 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 72 FOLLOWER]: Leader election lost for term 72. Reason: could not achieve majority
I20250113 21:10:15.704916 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 72 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 71. Current term is 72. Ops: []
I20250113 21:10:15.705152 28477 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 71.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250113 21:10:15.705281 28472 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 71 LEADER]: Stepping down as leader of term 71
I20250113 21:10:15.705328 28472 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 71 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:15.705389 28472 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 71.81, Last appended by leader: 81, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.705476 28472 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 71 FOLLOWER]: Advancing to term 72
I20250113 21:10:15.727432 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:15.802603 28476 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 71 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:15.802718 28476 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 71 FOLLOWER]: Advancing to term 72
I20250113 21:10:15.803406 28476 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.803579 28476 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 72 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:15.803797 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 72 candidate_status { last_received { term: 71 index: 81 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:15.803886 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 72: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:15.803874 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 72 candidate_status { last_received { term: 71 index: 81 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:15.804541 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 72.
I20250113 21:10:15.804705 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 72 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:15.804795 28476 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 72 FOLLOWER]: Leader election won for term 72
I20250113 21:10:15.804847 28476 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 72 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:15.804894 28476 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 71.81, Last appended by leader: 81, Current term: 72, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.805332 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 71 to 72, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 72 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:15.843716 28474 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 72 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:15.843818 28474 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 72 FOLLOWER]: Advancing to term 73
I20250113 21:10:15.844445 28474 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.844594 28474 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 73 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:15.844795 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 73 candidate_status { last_received { term: 71 index: 81 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:15.844795 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 73 candidate_status { last_received { term: 71 index: 81 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:15.844892 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 72 FOLLOWER]: Advancing to term 73
I20250113 21:10:15.845408 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 73.
I20250113 21:10:15.845582 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:15.845691 28474 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 73 FOLLOWER]: Leader election won for term 73
I20250113 21:10:15.845816 28474 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 73 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:15.845891 28474 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 71.81, Last appended by leader: 81, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.846324 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 72 to 73, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 73 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:15.880811 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 73 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 72. Current term is 73. Ops: []
I20250113 21:10:15.881075 28476 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250113 21:10:15.881199 28476 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 72 LEADER]: Stepping down as leader of term 72
I20250113 21:10:15.881235 28476 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 72 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:15.881307 28476 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 72.82, Last appended by leader: 82, Current term: 72, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:15.881379 28476 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 72 FOLLOWER]: Advancing to term 73
I20250113 21:10:15.934659 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 73 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 71 index: 81. Preceding OpId from leader: term: 73 index: 82. (index mismatch)
I20250113 21:10:15.934926 28474 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250113 21:10:15.935629 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 73 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 72 index: 82. Preceding OpId from leader: term: 73 index: 82. (term mismatch)
I20250113 21:10:15.935689 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 81
I20250113 21:10:15.935712 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 72.82
I20250113 21:10:15.935740 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 73 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:15.935892 28474 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250113 21:10:16.151937 28476 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 73 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:16.152053 28476 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 73 FOLLOWER]: Advancing to term 74
I20250113 21:10:16.152714 28476 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.152866 28476 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 74 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:16.153065 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 74 candidate_status { last_received { term: 73 index: 82 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:16.153071 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 74 candidate_status { last_received { term: 73 index: 82 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:16.153301 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 74 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:16.153384 28476 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 74 FOLLOWER]: Leader election lost for term 74. Reason: could not achieve majority
I20250113 21:10:16.163482 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 74 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 73. Current term is 74. Ops: []
I20250113 21:10:16.163705 28479 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 73.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250113 21:10:16.163882 28478 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 73 LEADER]: Stepping down as leader of term 73
I20250113 21:10:16.163928 28478 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 73 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:16.164017 28478 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 73.82, Last appended by leader: 82, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.164113 28478 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 73 FOLLOWER]: Advancing to term 74
I20250113 21:10:16.213165 28472 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 73 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:16.213265 28472 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 73 FOLLOWER]: Advancing to term 74
I20250113 21:10:16.213941 28472 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.214097 28472 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 74 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:16.214303 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 74 candidate_status { last_received { term: 73 index: 82 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:16.214319 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 74 candidate_status { last_received { term: 73 index: 82 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:16.214375 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 74 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 74: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:16.214989 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 74.
I20250113 21:10:16.215164 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 74 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:16.215266 28472 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 74 FOLLOWER]: Leader election won for term 74
I20250113 21:10:16.215363 28472 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 74 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:16.215418 28472 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 73.82, Last appended by leader: 82, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.215837 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 73 to 74, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 74 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:16.296880 28476 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 74 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:16.296986 28476 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 74 FOLLOWER]: Advancing to term 75
I20250113 21:10:16.297710 28476 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.297912 28476 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 75 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:16.298086 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 75 candidate_status { last_received { term: 73 index: 82 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:16.298091 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 75 candidate_status { last_received { term: 73 index: 82 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:16.298208 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 74 FOLLOWER]: Advancing to term 75
I20250113 21:10:16.298761 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 75 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 75.
I20250113 21:10:16.298921 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 75 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:16.299003 28476 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 75 FOLLOWER]: Leader election won for term 75
I20250113 21:10:16.299068 28476 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 75 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:16.299124 28476 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 73.82, Last appended by leader: 82, Current term: 75, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.299572 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 74 to 75, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 75 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:16.310267 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 75 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 74. Current term is 75. Ops: []
I20250113 21:10:16.310479 28472 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250113 21:10:16.310557 28472 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 75.83 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 82
I20250113 21:10:16.310601 28480 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 74 LEADER]: Stepping down as leader of term 74
I20250113 21:10:16.310639 28480 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 74 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:16.310695 28480 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 74.83, Last appended by leader: 83, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.310773 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 74 FOLLOWER]: Advancing to term 75
I20250113 21:10:16.406160 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 75 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 74 index: 83. Preceding OpId from leader: term: 75 index: 83. (term mismatch)
I20250113 21:10:16.406237 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 82
I20250113 21:10:16.406260 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 74.83
I20250113 21:10:16.406279 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 75 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:16.406481 28476 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250113 21:10:16.407418 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 75 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 73 index: 82. Preceding OpId from leader: term: 75 index: 83. (index mismatch)
I20250113 21:10:16.407605 28473 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250113 21:10:16.690349 28478 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 75 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:16.690448 28478 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 75 FOLLOWER]: Advancing to term 76
I20250113 21:10:16.691126 28478 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.691287 28478 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 76 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:16.691474 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 76 candidate_status { last_received { term: 75 index: 83 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:16.691504 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 76 candidate_status { last_received { term: 75 index: 83 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:16.691702 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 76 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:16.691779 28478 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 76 FOLLOWER]: Leader election lost for term 76. Reason: could not achieve majority
I20250113 21:10:16.703605 28480 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 75 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:16.703687 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 75 FOLLOWER]: Advancing to term 76
I20250113 21:10:16.704303 28480 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.704442 28480 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 76 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:16.704674 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 76 candidate_status { last_received { term: 75 index: 83 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:16.704695 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 76 candidate_status { last_received { term: 75 index: 83 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:16.704756 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 76: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:16.704905 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 76 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:16.704989 28480 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 76 FOLLOWER]: Leader election lost for term 76. Reason: could not achieve majority
I20250113 21:10:16.712146 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 76 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 75. Current term is 76. Ops: []
I20250113 21:10:16.712357 28481 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 75.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250113 21:10:16.712450 28481 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 75 LEADER]: Stepping down as leader of term 75
I20250113 21:10:16.712483 28481 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 75 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:16.712529 28481 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 75.83, Last appended by leader: 83, Current term: 75, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.712610 28481 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 75 FOLLOWER]: Advancing to term 76
I20250113 21:10:16.727689 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:16.807624 28478 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 76 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:16.807724 28478 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 76 FOLLOWER]: Advancing to term 77
I20250113 21:10:16.808429 28478 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.808614 28478 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 77 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:16.808806 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 77 candidate_status { last_received { term: 75 index: 83 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:16.808810 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 77 candidate_status { last_received { term: 75 index: 83 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:16.808888 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 76 FOLLOWER]: Advancing to term 77
I20250113 21:10:16.808921 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 76 FOLLOWER]: Advancing to term 77
I20250113 21:10:16.809396 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 77.
I20250113 21:10:16.809504 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 77.
I20250113 21:10:16.809571 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 77 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:16.809665 28478 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 77 FOLLOWER]: Leader election won for term 77
I20250113 21:10:16.809784 28478 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 77 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:16.809860 28478 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 75.83, Last appended by leader: 83, Current term: 77, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:16.810307 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 75 to 77, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 77 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:16.897223 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 77 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 75 index: 83. Preceding OpId from leader: term: 77 index: 84. (index mismatch)
I20250113 21:10:16.897564 28478 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250113 21:10:16.898695 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 77 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 75 index: 83. Preceding OpId from leader: term: 77 index: 84. (index mismatch)
I20250113 21:10:16.898923 28478 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250113 21:10:16.999200 28481 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 77 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:16.999310 28481 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 77 FOLLOWER]: Advancing to term 78
I20250113 21:10:16.999805 28480 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 77 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:16.999867 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 77 FOLLOWER]: Advancing to term 78
I20250113 21:10:17.000074 28481 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.000272 28481 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 78 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:17.000458 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 78 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.000464 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 78 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:17.000491 28480 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.000684 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 78 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 78: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:17.000741 28480 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 78 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:17.000857 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 78 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:17.000864 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 78 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:17.000928 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 78 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 78: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:17.000885 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 78 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.000994 28481 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 78 FOLLOWER]: Leader election lost for term 78. Reason: could not achieve majority
I20250113 21:10:17.001130 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 78 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:17.001221 28480 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 78 FOLLOWER]: Leader election lost for term 78. Reason: could not achieve majority
I20250113 21:10:17.014671 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 78 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 77. Current term is 78. Ops: []
I20250113 21:10:17.014907 28482 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 77.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250113 21:10:17.015034 28482 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 77 LEADER]: Stepping down as leader of term 77
I20250113 21:10:17.015069 28482 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 77 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:17.015131 28482 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 77.84, Last appended by leader: 84, Current term: 77, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.015215 28482 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 77 FOLLOWER]: Advancing to term 78
I20250113 21:10:17.128297 28480 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 78 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:17.128412 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 78 FOLLOWER]: Advancing to term 79
I20250113 21:10:17.129112 28480 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.129307 28480 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 79 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:17.129323 28481 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 78 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:17.129398 28481 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 78 FOLLOWER]: Advancing to term 79
I20250113 21:10:17.129503 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 79 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:17.129518 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 79 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.129587 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 78 FOLLOWER]: Advancing to term 79
I20250113 21:10:17.130240 28481 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.130240 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 79.
I20250113 21:10:17.130399 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 79: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:17.130443 28481 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 79 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:17.130504 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 79 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:17.130589 28480 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 79 FOLLOWER]: Leader election won for term 79
I20250113 21:10:17.130631 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 79 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:17.130651 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 79 candidate_status { last_received { term: 77 index: 84 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.130710 28480 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 79 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:17.130735 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 79: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:17.130759 28480 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 77.84, Last appended by leader: 84, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.131044 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:17.131167 28481 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20250113 21:10:17.131209 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 77 to 79, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 79 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:17.218714 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 79 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 77 index: 84. Preceding OpId from leader: term: 79 index: 85. (index mismatch)
I20250113 21:10:17.218968 28480 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250113 21:10:17.219905 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 79 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 77 index: 84. Preceding OpId from leader: term: 79 index: 85. (index mismatch)
I20250113 21:10:17.220122 28489 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250113 21:10:17.320899 28481 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 79 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:17.320997 28481 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 79 FOLLOWER]: Advancing to term 80
I20250113 21:10:17.321712 28481 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.321889 28481 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 80 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:17.322079 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 80 candidate_status { last_received { term: 79 index: 85 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:17.322079 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 80 candidate_status { last_received { term: 79 index: 85 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.322305 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 80 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:17.322382 28481 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 80 FOLLOWER]: Leader election lost for term 80. Reason: could not achieve majority
I20250113 21:10:17.335928 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 80 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 79. Current term is 80. Ops: []
I20250113 21:10:17.336215 28480 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 79.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250113 21:10:17.336333 28480 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 79 LEADER]: Stepping down as leader of term 79
I20250113 21:10:17.336364 28480 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 79 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:17.336421 28480 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 79.85, Last appended by leader: 85, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.336513 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 79 FOLLOWER]: Advancing to term 80
I20250113 21:10:17.410539 28482 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 79 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:17.410641 28482 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 79 FOLLOWER]: Advancing to term 80
I20250113 21:10:17.411334 28482 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.411553 28482 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 80 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:17.411729 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 80 candidate_status { last_received { term: 79 index: 85 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:17.411770 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 80 candidate_status { last_received { term: 79 index: 85 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:17.411809 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 80 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 80: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:17.412364 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 80.
I20250113 21:10:17.412590 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:17.412701 28482 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 80 FOLLOWER]: Leader election won for term 80
I20250113 21:10:17.412755 28482 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 80 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:17.412804 28482 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 79.85, Last appended by leader: 85, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.413223 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 79 to 80, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 80 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:17.457492 28481 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 80 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:17.457603 28481 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 80 FOLLOWER]: Advancing to term 81
I20250113 21:10:17.458321 28481 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.458482 28481 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 81 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:17.458700 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 81 candidate_status { last_received { term: 79 index: 85 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.458700 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 81 candidate_status { last_received { term: 79 index: 85 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:17.458850 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 80 FOLLOWER]: Advancing to term 81
I20250113 21:10:17.459442 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 81.
I20250113 21:10:17.459604 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 81 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:17.459697 28481 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 81 FOLLOWER]: Leader election won for term 81
I20250113 21:10:17.459848 28481 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 81 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:17.459895 28481 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 79.85, Last appended by leader: 85, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.460312 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 80 to 81, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 81 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:17.514756 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 81 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 80. Current term is 81. Ops: []
I20250113 21:10:17.515044 28482 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250113 21:10:17.515175 28482 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 80 LEADER]: Stepping down as leader of term 80
I20250113 21:10:17.515205 28482 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 80 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:17.515271 28482 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.86, Last appended by leader: 86, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.515342 28482 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 80 FOLLOWER]: Advancing to term 81
I20250113 21:10:17.552626 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 81 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 80 index: 86. Preceding OpId from leader: term: 81 index: 86. (term mismatch)
I20250113 21:10:17.552703 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 85
I20250113 21:10:17.552731 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 80.86
I20250113 21:10:17.552744 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 81 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:17.552964 28481 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250113 21:10:17.553966 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 81 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 79 index: 85. Preceding OpId from leader: term: 81 index: 86. (index mismatch)
I20250113 21:10:17.554153 28491 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250113 21:10:17.727973 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:17.745292 28480 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 81 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:17.745383 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 81 FOLLOWER]: Advancing to term 82
I20250113 21:10:17.746145 28480 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.746322 28480 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 82 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:17.746524 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 82 candidate_status { last_received { term: 81 index: 86 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:17.746551 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 82 candidate_status { last_received { term: 81 index: 86 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.746816 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 82 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:17.746958 28480 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 82 FOLLOWER]: Leader election lost for term 82. Reason: could not achieve majority
I20250113 21:10:17.747367 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 82 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 81. Current term is 82. Ops: []
I20250113 21:10:17.747560 28481 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 81.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250113 21:10:17.747658 28481 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 81 LEADER]: Stepping down as leader of term 81
I20250113 21:10:17.747700 28481 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 81 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:17.747753 28481 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 81.86, Last appended by leader: 86, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.747828 28481 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 81 FOLLOWER]: Advancing to term 82
I20250113 21:10:17.751302 28482 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 81 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:17.751350 28482 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 81 FOLLOWER]: Advancing to term 82
I20250113 21:10:17.751880 28482 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.752007 28482 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 82 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:17.752151 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 82 candidate_status { last_received { term: 81 index: 86 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:17.752162 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 82 candidate_status { last_received { term: 81 index: 86 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:17.752221 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 82 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 82: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:17.752691 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 82.
I20250113 21:10:17.752847 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 82 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:17.752923 28482 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 82 FOLLOWER]: Leader election won for term 82
I20250113 21:10:17.752974 28482 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 82 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:17.753026 28482 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 81.86, Last appended by leader: 86, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.753405 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 81 to 82, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 82 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:17.833813 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 82 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 81 index: 86. Preceding OpId from leader: term: 82 index: 87. (index mismatch)
I20250113 21:10:17.834136 28482 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250113 21:10:17.835086 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 82 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 81 index: 86. Preceding OpId from leader: term: 82 index: 87. (index mismatch)
I20250113 21:10:17.835290 28487 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250113 21:10:17.936619 28480 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 82 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:17.936733 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 82 FOLLOWER]: Advancing to term 83
I20250113 21:10:17.937393 28480 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.937598 28480 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 83 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:17.937836 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 83 candidate_status { last_received { term: 82 index: 87 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:17.937822 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 83 candidate_status { last_received { term: 82 index: 87 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:17.937983 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 82 FOLLOWER]: Advancing to term 83
I20250113 21:10:17.938529 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 83.
I20250113 21:10:17.938702 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:17.938809 28480 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 83 FOLLOWER]: Leader election won for term 83
I20250113 21:10:17.938948 28480 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 83 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:17.939018 28480 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 82.87, Last appended by leader: 87, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.939481 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 82 to 83, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 83 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:17.940454 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 83 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 82. Current term is 83. Ops: []
I20250113 21:10:17.940651 28492 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 82.87, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250113 21:10:17.940742 28492 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 83.88 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 87
I20250113 21:10:17.940790 28492 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 82 LEADER]: Stepping down as leader of term 82
I20250113 21:10:17.940815 28492 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 82 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:17.940865 28492 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 82.87, Last appended by leader: 87, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:17.940941 28492 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 82 FOLLOWER]: Advancing to term 83
I20250113 21:10:18.033885 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 83 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 82 index: 87. Preceding OpId from leader: term: 83 index: 88. (index mismatch)
I20250113 21:10:18.034145 28480 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250113 21:10:18.035063 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 83 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 82 index: 87. Preceding OpId from leader: term: 83 index: 88. (index mismatch)
I20250113 21:10:18.035247 28493 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250113 21:10:18.218628 28487 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 83 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:18.218722 28487 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 83 FOLLOWER]: Advancing to term 84
I20250113 21:10:18.219452 28487 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.219691 28487 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 84 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:18.219786 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 84 candidate_status { last_received { term: 83 index: 88 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:18.219792 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 84 candidate_status { last_received { term: 83 index: 88 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:18.220048 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 84 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:18.220134 28487 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 84 FOLLOWER]: Leader election lost for term 84. Reason: could not achieve majority
I20250113 21:10:18.228437 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 84 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 83. Current term is 84. Ops: []
I20250113 21:10:18.228662 28494 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 83.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250113 21:10:18.228785 28480 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 83 LEADER]: Stepping down as leader of term 83
I20250113 21:10:18.228837 28480 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 83 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:18.228909 28480 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 83.88, Last appended by leader: 88, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.228986 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 83 FOLLOWER]: Advancing to term 84
I20250113 21:10:18.318449 28490 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 83 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:18.318553 28490 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 83 FOLLOWER]: Advancing to term 84
I20250113 21:10:18.319320 28490 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.319494 28490 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 84 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:18.319722 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 84 candidate_status { last_received { term: 83 index: 88 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:18.319747 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 84 candidate_status { last_received { term: 83 index: 88 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:18.319944 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 84 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 84: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:18.320534 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 84.
I20250113 21:10:18.320705 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 84 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:18.320786 28490 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 84 FOLLOWER]: Leader election won for term 84
I20250113 21:10:18.320839 28490 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 84 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:18.320892 28490 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 83.88, Last appended by leader: 88, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.321352 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 83 to 84, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 84 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:18.327488 28487 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 84 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:18.327548 28487 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 84 FOLLOWER]: Advancing to term 85
I20250113 21:10:18.328083 28487 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.328205 28487 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 85 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:18.328342 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 85 candidate_status { last_received { term: 83 index: 88 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:18.328342 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 85 candidate_status { last_received { term: 83 index: 88 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:18.328446 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 84 FOLLOWER]: Advancing to term 85
I20250113 21:10:18.328966 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 85.
I20250113 21:10:18.329133 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:18.329224 28487 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 85 FOLLOWER]: Leader election won for term 85
I20250113 21:10:18.329370 28487 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 85 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:18.329448 28487 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 83.88, Last appended by leader: 88, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.329923 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 84 to 85, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 85 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:18.407467 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 85 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 83 index: 88. Preceding OpId from leader: term: 85 index: 89. (index mismatch)
I20250113 21:10:18.407730 28487 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250113 21:10:18.408843 28332 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 84 LEADER]: Stepping down as leader of term 84
I20250113 21:10:18.408900 28332 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 84 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:18.408965 28332 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 84.89, Last appended by leader: 89, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.409039 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 84 FOLLOWER]: Advancing to term 85
I20250113 21:10:18.409660 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 85 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 84 index: 89. Preceding OpId from leader: term: 85 index: 89. (term mismatch)
I20250113 21:10:18.409688 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 88
I20250113 21:10:18.409700 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 84.89
I20250113 21:10:18.409713 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 85 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:18.409904 28492 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250113 21:10:18.509188 28480 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 85 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:18.509286 28480 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 85 FOLLOWER]: Advancing to term 86
I20250113 21:10:18.509986 28480 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.510154 28480 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 86 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:18.510347 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 86 candidate_status { last_received { term: 85 index: 89 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:18.510413 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 85 FOLLOWER]: Advancing to term 86
I20250113 21:10:18.510357 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 86 candidate_status { last_received { term: 85 index: 89 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:18.510964 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 86 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 86.
I20250113 21:10:18.511118 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 86 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:18.511196 28480 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 86 FOLLOWER]: Leader election won for term 86
I20250113 21:10:18.511245 28480 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 86 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:18.511292 28480 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 85.89, Last appended by leader: 89, Current term: 86, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.511768 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 85 to 86, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 86 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:18.511919 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 86 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 85. Current term is 86. Ops: []
I20250113 21:10:18.512118 28492 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 85.89, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250113 21:10:18.512252 28492 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 86.90 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 89
I20250113 21:10:18.512355 28482 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 85 LEADER]: Stepping down as leader of term 85
I20250113 21:10:18.512391 28482 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 85 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:18.512461 28482 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 85.89, Last appended by leader: 89, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.512552 28482 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 85 FOLLOWER]: Advancing to term 86
I20250113 21:10:18.588105 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 86 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 85 index: 89. Preceding OpId from leader: term: 86 index: 90. (index mismatch)
I20250113 21:10:18.588388 28480 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250113 21:10:18.589578 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 86 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 85 index: 89. Preceding OpId from leader: term: 86 index: 90. (index mismatch)
I20250113 21:10:18.589756 28494 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250113 21:10:18.728233 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:18.809120 28482 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 86 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:18.809217 28482 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 86 FOLLOWER]: Advancing to term 87
I20250113 21:10:18.809966 28482 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.810127 28482 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 87 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:18.810302 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 87 candidate_status { last_received { term: 86 index: 90 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:18.810308 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 87 candidate_status { last_received { term: 86 index: 90 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:18.810523 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 87 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:18.810607 28482 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 87 FOLLOWER]: Leader election lost for term 87. Reason: could not achieve majority
I20250113 21:10:18.830806 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 87 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 86. Current term is 87. Ops: []
I20250113 21:10:18.831040 28480 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 86.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250113 21:10:18.831171 28493 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 86 LEADER]: Stepping down as leader of term 86
I20250113 21:10:18.831211 28493 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 86 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:18.831280 28493 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 86.90, Last appended by leader: 90, Current term: 86, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.831367 28493 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 86 FOLLOWER]: Advancing to term 87
I20250113 21:10:18.868557 28490 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 86 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:18.868661 28490 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 86 FOLLOWER]: Advancing to term 87
I20250113 21:10:18.869341 28490 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.869524 28490 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 87 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:18.869711 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 87 candidate_status { last_received { term: 86 index: 90 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:18.869773 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 87: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:18.869729 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 87 candidate_status { last_received { term: 86 index: 90 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:18.870337 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 87.
I20250113 21:10:18.870498 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 87 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:18.870581 28490 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 87 FOLLOWER]: Leader election won for term 87
I20250113 21:10:18.870695 28490 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 87 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:18.870748 28490 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 86.90, Last appended by leader: 90, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.871181 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 86 to 87, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 87 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:18.916965 28482 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 87 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:18.917061 28482 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 87 FOLLOWER]: Advancing to term 88
I20250113 21:10:18.917726 28482 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.917914 28482 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 88 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:18.918079 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 88 candidate_status { last_received { term: 86 index: 90 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:18.918145 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 87 FOLLOWER]: Advancing to term 88
I20250113 21:10:18.918083 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 88 candidate_status { last_received { term: 86 index: 90 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:18.918699 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 88.
I20250113 21:10:18.918843 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 88 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:18.918916 28482 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 88 FOLLOWER]: Leader election won for term 88
I20250113 21:10:18.918967 28482 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 88 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:18.919018 28482 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 86.90, Last appended by leader: 90, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.919422 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 87 to 88, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 88 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:18.951472 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 88 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 87. Current term is 88. Ops: []
I20250113 21:10:18.951742 28490 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250113 21:10:18.951849 28490 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 88.91 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 90
I20250113 21:10:18.951872 28490 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 87 LEADER]: Stepping down as leader of term 87
I20250113 21:10:18.951885 28490 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 87 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:18.951932 28490 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 87.91, Last appended by leader: 91, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:18.952004 28490 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 87 FOLLOWER]: Advancing to term 88
I20250113 21:10:18.995610 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 88 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 86 index: 90. Preceding OpId from leader: term: 88 index: 91. (index mismatch)
I20250113 21:10:18.995877 28482 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250113 21:10:18.996929 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 88 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 87 index: 91. Preceding OpId from leader: term: 88 index: 91. (term mismatch)
I20250113 21:10:18.996984 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 90
I20250113 21:10:18.997002 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 87.91
I20250113 21:10:18.997025 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 88 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:18.997184 28492 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250113 21:10:19.191313 28493 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 88 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:19.191419 28493 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 88 FOLLOWER]: Advancing to term 89
I20250113 21:10:19.192107 28493 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.192276 28493 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 89 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:19.192502 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 89 candidate_status { last_received { term: 88 index: 91 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:19.192518 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 89 candidate_status { last_received { term: 88 index: 91 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:19.192772 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 89 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:19.192847 28493 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 89 FOLLOWER]: Leader election lost for term 89. Reason: could not achieve majority
I20250113 21:10:19.209167 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 89 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 88. Current term is 89. Ops: []
I20250113 21:10:19.209412 28496 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 88.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250113 21:10:19.209545 28496 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 88 LEADER]: Stepping down as leader of term 88
I20250113 21:10:19.209578 28496 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 88 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:19.209640 28496 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 88.91, Last appended by leader: 91, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.209738 28496 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 88 FOLLOWER]: Advancing to term 89
I20250113 21:10:19.276726 28490 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 88 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:19.276849 28490 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 88 FOLLOWER]: Advancing to term 89
I20250113 21:10:19.277534 28490 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.277704 28490 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 89 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:19.277909 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 89 candidate_status { last_received { term: 88 index: 91 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:19.277909 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 89 candidate_status { last_received { term: 88 index: 91 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:19.278024 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 89 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 89: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:19.278561 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 89.
I20250113 21:10:19.278712 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 89 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:19.278791 28490 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 89 FOLLOWER]: Leader election won for term 89
I20250113 21:10:19.278842 28490 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 89 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:19.278894 28490 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 88.91, Last appended by leader: 91, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.279362 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 88 to 89, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 89 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:19.331671 28493 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 89 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:19.331772 28493 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 89 FOLLOWER]: Advancing to term 90
I20250113 21:10:19.332409 28493 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.332594 28493 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 90 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:19.332818 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 90 candidate_status { last_received { term: 88 index: 91 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:19.332818 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 90 candidate_status { last_received { term: 88 index: 91 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:19.332948 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 89 FOLLOWER]: Advancing to term 90
I20250113 21:10:19.333532 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 90.
I20250113 21:10:19.333721 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 90 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:19.333802 28493 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 90 FOLLOWER]: Leader election won for term 90
I20250113 21:10:19.333936 28493 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 90 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:19.333984 28493 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 88.91, Last appended by leader: 91, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.334407 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 89 to 90, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 90 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:19.354501 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 90 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 89. Current term is 90. Ops: []
I20250113 21:10:19.354748 28490 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250113 21:10:19.354835 28490 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 90.92 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 91
I20250113 21:10:19.354876 28495 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 89 LEADER]: Stepping down as leader of term 89
I20250113 21:10:19.354918 28495 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 89 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:19.354976 28495 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 89.92, Last appended by leader: 92, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.355054 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 89 FOLLOWER]: Advancing to term 90
I20250113 21:10:19.424885 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 90 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 88 index: 91. Preceding OpId from leader: term: 90 index: 92. (index mismatch)
I20250113 21:10:19.425207 28493 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250113 21:10:19.426330 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 90 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 89 index: 92. Preceding OpId from leader: term: 90 index: 92. (term mismatch)
I20250113 21:10:19.426393 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 91
I20250113 21:10:19.426412 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 89.92
I20250113 21:10:19.426436 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 90 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:19.426614 28497 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250113 21:10:19.719995 28495 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 90 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:19.720095 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 90 FOLLOWER]: Advancing to term 91
I20250113 21:10:19.720788 28495 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.720950 28495 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 91 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:19.721145 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 91 candidate_status { last_received { term: 90 index: 92 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:19.721159 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 91 candidate_status { last_received { term: 90 index: 92 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:19.721398 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 91 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:19.721510 28495 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 91 FOLLOWER]: Leader election lost for term 91. Reason: could not achieve majority
I20250113 21:10:19.728500 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:19.731600 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 91 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 90. Current term is 91. Ops: []
I20250113 21:10:19.731823 28498 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 90.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250113 21:10:19.731948 28493 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 90 LEADER]: Stepping down as leader of term 90
I20250113 21:10:19.732007 28493 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 90 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:19.732069 28493 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 90.92, Last appended by leader: 92, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.732151 28493 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 90 FOLLOWER]: Advancing to term 91
I20250113 21:10:19.780020 28496 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 90 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:19.780112 28496 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 90 FOLLOWER]: Advancing to term 91
I20250113 21:10:19.780822 28496 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.780999 28496 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 91 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:19.781203 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 91 candidate_status { last_received { term: 90 index: 92 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:19.781224 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 91 candidate_status { last_received { term: 90 index: 92 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:19.781272 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 91 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 91: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:19.781940 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 91 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 91.
I20250113 21:10:19.782106 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 91 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:19.782193 28496 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 91 FOLLOWER]: Leader election won for term 91
I20250113 21:10:19.782339 28496 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 91 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:19.782393 28496 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 90.92, Last appended by leader: 92, Current term: 91, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.782855 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 90 to 91, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 91 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:19.829627 28495 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 91 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:19.829737 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 91 FOLLOWER]: Advancing to term 92
I20250113 21:10:19.830439 28495 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.830610 28495 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 92 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:19.830796 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 92 candidate_status { last_received { term: 90 index: 92 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:19.830796 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 92 candidate_status { last_received { term: 90 index: 92 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:19.830917 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 91 FOLLOWER]: Advancing to term 92
I20250113 21:10:19.831487 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 92 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 92.
I20250113 21:10:19.831643 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 92 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:19.831738 28495 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 92 FOLLOWER]: Leader election won for term 92
I20250113 21:10:19.831790 28495 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 92 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:19.831838 28495 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 90.92, Last appended by leader: 92, Current term: 92, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.832283 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 91 to 92, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 92 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:19.883196 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 92 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 91. Current term is 92. Ops: []
I20250113 21:10:19.883440 28496 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250113 21:10:19.883560 28496 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 91 LEADER]: Stepping down as leader of term 91
I20250113 21:10:19.883594 28496 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 91 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:19.883661 28496 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 91.93, Last appended by leader: 93, Current term: 91, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:19.883729 28496 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 91 FOLLOWER]: Advancing to term 92
I20250113 21:10:19.917729 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 92 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 91 index: 93. Preceding OpId from leader: term: 92 index: 93. (term mismatch)
I20250113 21:10:19.917810 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 92
I20250113 21:10:19.917824 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 91.93
I20250113 21:10:19.917840 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 92 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:19.918063 28495 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250113 21:10:19.918987 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 92 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 90 index: 92. Preceding OpId from leader: term: 92 index: 93. (index mismatch)
I20250113 21:10:19.919185 28500 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250113 21:10:20.020006 28498 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 92 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:20.020090 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 92 FOLLOWER]: Advancing to term 93
I20250113 21:10:20.020781 28498 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.020920 28498 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 93 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:20.021113 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 93 candidate_status { last_received { term: 92 index: 93 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:20.021211 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 93 candidate_status { last_received { term: 92 index: 93 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:20.021440 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 93 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:20.021544 28498 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 93 FOLLOWER]: Leader election lost for term 93. Reason: could not achieve majority
I20250113 21:10:20.025835 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 93 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 92. Current term is 93. Ops: []
I20250113 21:10:20.026062 28495 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 92.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250113 21:10:20.026163 28495 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 92 LEADER]: Stepping down as leader of term 92
I20250113 21:10:20.026189 28495 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 92 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:20.026238 28495 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 92.93, Last appended by leader: 93, Current term: 92, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.026310 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 92 FOLLOWER]: Advancing to term 93
I20250113 21:10:20.118925 28496 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 92 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:20.119025 28496 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 92 FOLLOWER]: Advancing to term 93
I20250113 21:10:20.119701 28496 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.119875 28496 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 93 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:20.120118 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 93 candidate_status { last_received { term: 92 index: 93 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:20.120129 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 93 candidate_status { last_received { term: 92 index: 93 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:20.120239 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 93: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:20.120764 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 93.
I20250113 21:10:20.120929 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 93 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:20.121012 28496 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 93 FOLLOWER]: Leader election won for term 93
I20250113 21:10:20.121078 28496 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 93 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:20.121134 28496 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 92.93, Last appended by leader: 93, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.121555 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 92 to 93, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 93 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:20.136826 28498 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 93 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:20.136896 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 93 FOLLOWER]: Advancing to term 94
I20250113 21:10:20.137494 28498 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.137637 28498 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 94 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:20.137786 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 94 candidate_status { last_received { term: 92 index: 93 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:20.137840 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 93 FOLLOWER]: Advancing to term 94
I20250113 21:10:20.137801 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 94 candidate_status { last_received { term: 92 index: 93 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:20.138423 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 94 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 94.
I20250113 21:10:20.138576 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 94 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:20.138664 28498 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 94 FOLLOWER]: Leader election won for term 94
I20250113 21:10:20.138717 28498 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 94 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:20.138763 28498 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 92.93, Last appended by leader: 93, Current term: 94, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.139173 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 93 to 94, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 94 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:20.215606 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 94 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 92 index: 93. Preceding OpId from leader: term: 94 index: 94. (index mismatch)
I20250113 21:10:20.215857 28498 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250113 21:10:20.216873 28256 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 93 LEADER]: Stepping down as leader of term 93
I20250113 21:10:20.216938 28256 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 93 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:20.216989 28256 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 93.94, Last appended by leader: 94, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.217063 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 93 FOLLOWER]: Advancing to term 94
I20250113 21:10:20.217720 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 94 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 93 index: 94. Preceding OpId from leader: term: 94 index: 94. (term mismatch)
I20250113 21:10:20.217759 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 93
I20250113 21:10:20.217772 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 93.94
I20250113 21:10:20.217784 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 94 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:20.217993 28501 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250113 21:10:20.531771 28496 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 94 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:20.531890 28496 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 94 FOLLOWER]: Advancing to term 95
I20250113 21:10:20.532569 28496 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.532754 28496 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 95 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:20.533061 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 95 candidate_status { last_received { term: 94 index: 94 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:20.533061 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 95 candidate_status { last_received { term: 94 index: 94 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:20.533349 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 95 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:20.533459 28496 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 95 FOLLOWER]: Leader election lost for term 95. Reason: could not achieve majority
I20250113 21:10:20.535322 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 95 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 94. Current term is 95. Ops: []
I20250113 21:10:20.535573 28498 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 94.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250113 21:10:20.535666 28498 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 94 LEADER]: Stepping down as leader of term 94
I20250113 21:10:20.535702 28498 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 94 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:20.535751 28498 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 94.94, Last appended by leader: 94, Current term: 94, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.535845 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 94 FOLLOWER]: Advancing to term 95
I20250113 21:10:20.559175 28495 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 94 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:20.559283 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 94 FOLLOWER]: Advancing to term 95
I20250113 21:10:20.559980 28495 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.560153 28495 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 95 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:20.560415 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 95 candidate_status { last_received { term: 94 index: 94 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:20.560418 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 95 candidate_status { last_received { term: 94 index: 94 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:20.560607 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 95: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:20.561116 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 95.
I20250113 21:10:20.561354 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 95 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:20.561501 28495 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 95 FOLLOWER]: Leader election won for term 95
I20250113 21:10:20.561723 28495 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 95 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:20.561791 28495 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 94.94, Last appended by leader: 94, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.562253 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 94 to 95, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 95 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:20.655628 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 95 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 94 index: 94. Preceding OpId from leader: term: 95 index: 95. (index mismatch)
I20250113 21:10:20.655773 28496 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 95 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:20.655858 28496 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 95 FOLLOWER]: Advancing to term 96
I20250113 21:10:20.655925 28495 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250113 21:10:20.656600 28496 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.656725 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 96 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 95. Current term is 96. Ops: [95.95-95.95]
I20250113 21:10:20.656754 28496 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 96 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:20.656884 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 96 candidate_status { last_received { term: 94 index: 94 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:20.656908 28495 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 94.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250113 21:10:20.656997 28495 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 95 LEADER]: Stepping down as leader of term 95
I20250113 21:10:20.657032 28495 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 95 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:20.657083 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 96 candidate_status { last_received { term: 94 index: 94 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:20.657147 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 95 FOLLOWER]: Advancing to term 96
I20250113 21:10:20.657083 28495 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 95.95, Last appended by leader: 95, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.657222 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 95 FOLLOWER]: Advancing to term 96
I20250113 21:10:20.657719 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 96.
I20250113 21:10:20.657838 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 96 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:20.657943 28496 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 96 FOLLOWER]: Leader election won for term 96
I20250113 21:10:20.658054 28496 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 96 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:20.658103 28496 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 94.94, Last appended by leader: 94, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.658584 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 95 to 96, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 96 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:20.728710 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:20.735795 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 96 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 95 index: 95. Preceding OpId from leader: term: 96 index: 95. (term mismatch)
I20250113 21:10:20.735865 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 94
I20250113 21:10:20.735878 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 95.95
I20250113 21:10:20.735893 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 96 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:20.736089 28496 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250113 21:10:20.736984 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 96 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 94 index: 94. Preceding OpId from leader: term: 96 index: 95. (index mismatch)
I20250113 21:10:20.737201 28503 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250113 21:10:20.838096 28501 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 96 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:20.838191 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 96 FOLLOWER]: Advancing to term 97
I20250113 21:10:20.839021 28501 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.839187 28501 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 97 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:20.839398 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 97 candidate_status { last_received { term: 96 index: 95 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:20.839404 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 97 candidate_status { last_received { term: 96 index: 95 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:20.839607 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 97 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:20.839710 28501 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 97 FOLLOWER]: Leader election lost for term 97. Reason: could not achieve majority
I20250113 21:10:20.850066 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 97 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 96. Current term is 97. Ops: []
I20250113 21:10:20.850317 28504 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 96.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250113 21:10:20.850431 28504 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 96 LEADER]: Stepping down as leader of term 96
I20250113 21:10:20.850467 28504 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 96 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:20.850525 28504 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 96.95, Last appended by leader: 95, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.850602 28504 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 96 FOLLOWER]: Advancing to term 97
I20250113 21:10:20.929961 28495 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 96 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:20.930094 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 96 FOLLOWER]: Advancing to term 97
I20250113 21:10:20.930766 28495 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.930938 28495 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 97 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:20.931119 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 97 candidate_status { last_received { term: 96 index: 95 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:20.931169 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 97 candidate_status { last_received { term: 96 index: 95 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:20.931269 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 97 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 97: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:20.931721 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 97 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 97.
I20250113 21:10:20.931931 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 97 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:20.932030 28495 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 97 FOLLOWER]: Leader election won for term 97
I20250113 21:10:20.932098 28495 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 97 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:20.932163 28495 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 96.95, Last appended by leader: 95, Current term: 97, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.932585 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 96 to 97, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 97 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:20.961527 28501 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 97 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:20.961642 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 97 FOLLOWER]: Advancing to term 98
I20250113 21:10:20.962345 28501 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.962555 28501 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 98 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:20.962752 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 98 candidate_status { last_received { term: 96 index: 95 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:20.962750 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 98 candidate_status { last_received { term: 96 index: 95 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:20.962870 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 97 FOLLOWER]: Advancing to term 98
I20250113 21:10:20.963413 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 98.
I20250113 21:10:20.963629 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 98 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:20.963735 28501 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 98 FOLLOWER]: Leader election won for term 98
I20250113 21:10:20.963785 28501 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 98 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:20.963856 28501 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 96.95, Last appended by leader: 95, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:20.964419 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 97 to 98, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 98 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:21.038084 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 98 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 97. Current term is 98. Ops: []
I20250113 21:10:21.038352 28495 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250113 21:10:21.038477 28495 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 97 LEADER]: Stepping down as leader of term 97
I20250113 21:10:21.038513 28495 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 97 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:21.038619 28495 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 97.96, Last appended by leader: 96, Current term: 97, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:21.038691 28495 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 97 FOLLOWER]: Advancing to term 98
I20250113 21:10:21.055820 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 98 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 97 index: 96. Preceding OpId from leader: term: 98 index: 96. (term mismatch)
I20250113 21:10:21.055895 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 95
I20250113 21:10:21.055909 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 97.96
I20250113 21:10:21.055924 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 98 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:21.056104 28501 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250113 21:10:21.056977 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 98 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 96 index: 95. Preceding OpId from leader: term: 98 index: 96. (index mismatch)
I20250113 21:10:21.057176 28498 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250113 21:10:21.610026 28505 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 98 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:21.610134 28505 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 98 FOLLOWER]: Advancing to term 99
I20250113 21:10:21.610896 28505 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:21.611068 28505 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 99 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:21.611272 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 99 candidate_status { last_received { term: 98 index: 96 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:21.611277 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 99 candidate_status { last_received { term: 98 index: 96 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:21.611539 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:21.611617 28505 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20250113 21:10:21.618467 28506 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 98 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:21.618535 28506 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 98 FOLLOWER]: Advancing to term 99
I20250113 21:10:21.619267 28506 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:21.619421 28506 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 99 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:21.619580 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 99 candidate_status { last_received { term: 98 index: 96 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:21.619596 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 99 candidate_status { last_received { term: 98 index: 96 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:21.619665 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 99: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:21.619804 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:21.619884 28506 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20250113 21:10:21.627171 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 99 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 98. Current term is 99. Ops: []
I20250113 21:10:21.627362 28498 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 98.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250113 21:10:21.627457 28498 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 98 LEADER]: Stepping down as leader of term 98
I20250113 21:10:21.627493 28498 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 98 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:21.627552 28498 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 98.96, Last appended by leader: 96, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:21.627619 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 98 FOLLOWER]: Advancing to term 99
I20250113 21:10:21.726667 28505 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 99 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:21.726778 28505 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 99 FOLLOWER]: Advancing to term 100
I20250113 21:10:21.727494 28505 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:21.727670 28505 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 100 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:21.727864 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 100 candidate_status { last_received { term: 98 index: 96 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:21.727933 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 99 FOLLOWER]: Advancing to term 100
I20250113 21:10:21.727874 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 100 candidate_status { last_received { term: 98 index: 96 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:21.728035 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 99 FOLLOWER]: Advancing to term 100
I20250113 21:10:21.728566 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 100.
I20250113 21:10:21.728615 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 100.
I20250113 21:10:21.728766 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 100 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:21.728859 28505 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 100 FOLLOWER]: Leader election won for term 100
I20250113 21:10:21.728963 28505 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 100 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:21.729007 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:21.729007 28505 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 98.96, Last appended by leader: 96, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:21.729465 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 98 to 100, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 100 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:21.807641 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 100 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 98 index: 96. Preceding OpId from leader: term: 100 index: 97. (index mismatch)
I20250113 21:10:21.807920 28505 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250113 21:10:21.809139 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 100 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 98 index: 96. Preceding OpId from leader: term: 100 index: 97. (index mismatch)
I20250113 21:10:21.809322 28511 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250113 21:10:21.996140 28498 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 100 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:21.996240 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 100 FOLLOWER]: Advancing to term 101
I20250113 21:10:21.996907 28498 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:21.997066 28498 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 101 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:21.997275 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 101 candidate_status { last_received { term: 100 index: 97 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:21.997279 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 101 candidate_status { last_received { term: 100 index: 97 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:21.997522 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 101 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:21.997629 28498 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 101 FOLLOWER]: Leader election lost for term 101. Reason: could not achieve majority
I20250113 21:10:22.008260 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 101 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 100. Current term is 101. Ops: []
I20250113 21:10:22.008527 28511 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 100.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250113 21:10:22.008646 28511 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 100 LEADER]: Stepping down as leader of term 100
I20250113 21:10:22.008680 28511 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 100 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:22.008744 28511 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 100.97, Last appended by leader: 97, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.008837 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 100 FOLLOWER]: Advancing to term 101
I20250113 21:10:22.071162 28506 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 100 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:22.071259 28506 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 100 FOLLOWER]: Advancing to term 101
I20250113 21:10:22.072012 28506 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.072185 28506 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 101 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:22.072389 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 101 candidate_status { last_received { term: 100 index: 97 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:22.072410 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 101 candidate_status { last_received { term: 100 index: 97 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:22.072536 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 101 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 101: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:22.073282 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 101 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 101.
I20250113 21:10:22.073463 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 101 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:22.073554 28506 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 101 FOLLOWER]: Leader election won for term 101
I20250113 21:10:22.073707 28506 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 101 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:22.073761 28506 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 100.97, Last appended by leader: 97, Current term: 101, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.074235 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 100 to 101, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 101 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:22.122223 28498 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 101 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:22.122320 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 101 FOLLOWER]: Advancing to term 102
I20250113 21:10:22.122984 28498 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.123143 28498 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 102 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:22.123351 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 102 candidate_status { last_received { term: 100 index: 97 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:22.123351 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 102 candidate_status { last_received { term: 100 index: 97 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:22.123466 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 101 FOLLOWER]: Advancing to term 102
I20250113 21:10:22.124140 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 102.
I20250113 21:10:22.124336 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 102 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:22.124440 28498 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 102 FOLLOWER]: Leader election won for term 102
I20250113 21:10:22.124495 28498 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 102 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:22.124548 28498 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 100.97, Last appended by leader: 97, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.125046 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 101 to 102, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 102 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:22.185341 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 102 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 101. Current term is 102. Ops: []
I20250113 21:10:22.185578 28506 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250113 21:10:22.185688 28506 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 101 LEADER]: Stepping down as leader of term 101
I20250113 21:10:22.185719 28506 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 101 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:22.185808 28506 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 101.98, Last appended by leader: 98, Current term: 101, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.185892 28506 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 101 FOLLOWER]: Advancing to term 102
I20250113 21:10:22.218358 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 102 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 100 index: 97. Preceding OpId from leader: term: 102 index: 98. (index mismatch)
I20250113 21:10:22.218652 28498 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250113 21:10:22.219662 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 102 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 101 index: 98. Preceding OpId from leader: term: 102 index: 98. (term mismatch)
I20250113 21:10:22.219723 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 97
I20250113 21:10:22.219743 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 101.98
I20250113 21:10:22.219763 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 102 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:22.219929 28498 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250113 21:10:22.398574 28511 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 102 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:22.398685 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 102 FOLLOWER]: Advancing to term 103
I20250113 21:10:22.399350 28511 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.399518 28511 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 103 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:22.399721 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:22.399734 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:22.399966 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 103 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:22.400059 28511 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 103 FOLLOWER]: Leader election lost for term 103. Reason: could not achieve majority
I20250113 21:10:22.413998 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 103 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 102. Current term is 103. Ops: []
I20250113 21:10:22.414260 28501 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 102.98, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250113 21:10:22.414388 28501 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 102 LEADER]: Stepping down as leader of term 102
I20250113 21:10:22.414417 28501 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 102 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:22.414474 28501 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 102.98, Last appended by leader: 98, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.414558 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 102 FOLLOWER]: Advancing to term 103
I20250113 21:10:22.430337 28512 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 102 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:22.430441 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 102 FOLLOWER]: Advancing to term 103
I20250113 21:10:22.431195 28512 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.431385 28512 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 103 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:22.431578 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:22.431649 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 103: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:22.431613 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:22.432219 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 103.
I20250113 21:10:22.432394 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 103 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:22.432485 28512 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 103 FOLLOWER]: Leader election won for term 103
I20250113 21:10:22.432543 28512 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 103 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:22.432691 28512 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 102.98, Last appended by leader: 98, Current term: 103, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.433144 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 102 to 103, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 103 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:22.505123 28511 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:22.505239 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 103 FOLLOWER]: Advancing to term 104
I20250113 21:10:22.505903 28511 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.506057 28511 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 104 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:22.506274 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 104 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:22.506274 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 104 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:22.506414 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 103 FOLLOWER]: Advancing to term 104
I20250113 21:10:22.506976 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 104.
I20250113 21:10:22.507120 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:22.507198 28511 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 104 FOLLOWER]: Leader election won for term 104
I20250113 21:10:22.507264 28511 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 104 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:22.507316 28511 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 102.98, Last appended by leader: 98, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.507761 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 103 to 104, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 104 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:22.535148 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 104 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 103. Current term is 104. Ops: []
I20250113 21:10:22.535400 28512 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250113 21:10:22.535516 28512 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 103 LEADER]: Stepping down as leader of term 103
I20250113 21:10:22.535534 28512 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 103 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:22.535586 28512 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 103.99, Last appended by leader: 99, Current term: 103, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.535666 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 103 FOLLOWER]: Advancing to term 104
I20250113 21:10:22.600546 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 104 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 103 index: 99. Preceding OpId from leader: term: 104 index: 99. (term mismatch)
I20250113 21:10:22.600618 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 98
I20250113 21:10:22.600636 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 103.99
I20250113 21:10:22.600657 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 104 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:22.600934 28511 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250113 21:10:22.602048 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 104 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 102 index: 98. Preceding OpId from leader: term: 104 index: 99. (index mismatch)
I20250113 21:10:22.602262 28511 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250113 21:10:22.732507 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:22.794993 28512 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 104 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:22.795115 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 104 FOLLOWER]: Advancing to term 105
I20250113 21:10:22.795794 28512 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.795964 28512 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 105 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:22.796159 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 105 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:22.796159 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 105 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:22.796468 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:22.796603 28512 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20250113 21:10:22.799984 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 105 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 104. Current term is 105. Ops: []
I20250113 21:10:22.800216 28513 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 104.99, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250113 21:10:22.800343 28511 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 104 LEADER]: Stepping down as leader of term 104
I20250113 21:10:22.800380 28511 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:22.800439 28511 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 104.99, Last appended by leader: 99, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.800513 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 104 FOLLOWER]: Advancing to term 105
I20250113 21:10:22.856778 28498 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 104 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:22.856884 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 104 FOLLOWER]: Advancing to term 105
I20250113 21:10:22.857663 28498 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.857848 28498 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 105 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:22.858073 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 105 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:22.858083 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 105 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:22.858150 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 105: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:22.858716 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 105 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 105.
I20250113 21:10:22.858871 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 105 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:22.858979 28498 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 105 FOLLOWER]: Leader election won for term 105
I20250113 21:10:22.859035 28498 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 105 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:22.859091 28498 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 104.99, Last appended by leader: 99, Current term: 105, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.859498 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 104 to 105, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 105 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:22.903045 28512 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:22.903148 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 105 FOLLOWER]: Advancing to term 106
I20250113 21:10:22.903828 28512 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.904024 28512 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 106 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:22.904247 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 106 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:22.904247 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 106 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:22.904382 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 105 FOLLOWER]: Advancing to term 106
I20250113 21:10:22.904958 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 106 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 106.
I20250113 21:10:22.905134 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 106 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:22.905216 28512 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 106 FOLLOWER]: Leader election won for term 106
I20250113 21:10:22.905267 28512 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 106 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:22.905320 28512 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 104.99, Last appended by leader: 99, Current term: 106, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.905750 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 105 to 106, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 106 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:22.955907 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 106 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 105. Current term is 106. Ops: []
I20250113 21:10:22.956154 28498 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250113 21:10:22.956251 28498 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 105 LEADER]: Stepping down as leader of term 105
I20250113 21:10:22.956284 28498 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 105 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:22.956339 28498 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 105.100, Last appended by leader: 100, Current term: 105, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:22.956426 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 105 FOLLOWER]: Advancing to term 106
I20250113 21:10:22.980959 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 106 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 105 index: 100. Preceding OpId from leader: term: 106 index: 100. (term mismatch)
I20250113 21:10:22.981032 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 99
I20250113 21:10:22.981045 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 105.100
I20250113 21:10:22.981060 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 106 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:22.981273 28512 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250113 21:10:22.982208 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 106 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 104 index: 99. Preceding OpId from leader: term: 106 index: 100. (index mismatch)
I20250113 21:10:22.982381 28512 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250113 21:10:23.190953 28498 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 106 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:23.191058 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 106 FOLLOWER]: Advancing to term 107
I20250113 21:10:23.191764 28498 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.191929 28498 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 107 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:23.192166 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 107 candidate_status { last_received { term: 106 index: 100 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:23.192166 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 107 candidate_status { last_received { term: 106 index: 100 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:23.192431 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 107 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:23.192518 28498 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 107 FOLLOWER]: Leader election lost for term 107. Reason: could not achieve majority
I20250113 21:10:23.208784 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 107 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 106. Current term is 107. Ops: []
I20250113 21:10:23.209110 28512 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 106.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250113 21:10:23.209249 28512 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 106 LEADER]: Stepping down as leader of term 106
I20250113 21:10:23.209282 28512 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 106 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:23.209362 28512 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 106.100, Last appended by leader: 100, Current term: 106, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.209439 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 106 FOLLOWER]: Advancing to term 107
I20250113 21:10:23.267592 28511 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 106 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:23.267693 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 106 FOLLOWER]: Advancing to term 107
I20250113 21:10:23.268317 28511 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.268605 28511 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 107 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:23.268769 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 107 candidate_status { last_received { term: 106 index: 100 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:23.268733 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 107 candidate_status { last_received { term: 106 index: 100 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:23.268857 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 107: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:23.269364 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 107.
I20250113 21:10:23.269536 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 107 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:23.269637 28511 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 107 FOLLOWER]: Leader election won for term 107
I20250113 21:10:23.269686 28511 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 107 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:23.269726 28511 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 106.100, Last appended by leader: 100, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.270216 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 106 to 107, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 107 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:23.322077 28498 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 107 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:23.322176 28498 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 107 FOLLOWER]: Advancing to term 108
I20250113 21:10:23.322813 28498 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.322985 28498 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 108 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:23.323199 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 108 candidate_status { last_received { term: 106 index: 100 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:23.323199 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 108 candidate_status { last_received { term: 106 index: 100 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:23.323302 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 107 FOLLOWER]: Advancing to term 108
I20250113 21:10:23.323853 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 108 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 108.
I20250113 21:10:23.324023 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 108 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:23.324132 28498 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 108 FOLLOWER]: Leader election won for term 108
I20250113 21:10:23.324194 28498 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 108 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:23.324255 28498 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 106.100, Last appended by leader: 100, Current term: 108, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.324698 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 107 to 108, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 108 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:23.354421 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 108 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 107. Current term is 108. Ops: []
I20250113 21:10:23.354671 28511 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250113 21:10:23.354792 28511 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 107 LEADER]: Stepping down as leader of term 107
I20250113 21:10:23.354827 28511 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 107 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:23.354892 28511 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 107.101, Last appended by leader: 101, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.354956 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 107 FOLLOWER]: Advancing to term 108
I20250113 21:10:23.421330 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 108 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 106 index: 100. Preceding OpId from leader: term: 108 index: 101. (index mismatch)
I20250113 21:10:23.421618 28498 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250113 21:10:23.422757 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 108 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 107 index: 101. Preceding OpId from leader: term: 108 index: 101. (term mismatch)
I20250113 21:10:23.422817 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 100
I20250113 21:10:23.422838 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 107.101
I20250113 21:10:23.422869 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:23.423041 28498 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250113 21:10:23.523238 28512 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 108 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:23.523346 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 108 FOLLOWER]: Advancing to term 109
I20250113 21:10:23.524004 28512 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.524152 28512 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 109 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:23.524370 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 109 candidate_status { last_received { term: 108 index: 101 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:23.524392 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 109 candidate_status { last_received { term: 108 index: 101 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:23.524648 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 109 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:23.524762 28512 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 109 FOLLOWER]: Leader election lost for term 109. Reason: could not achieve majority
I20250113 21:10:23.527529 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 109 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 108. Current term is 109. Ops: []
I20250113 21:10:23.527761 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 108.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250113 21:10:23.527877 28501 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 108 LEADER]: Stepping down as leader of term 108
I20250113 21:10:23.527915 28501 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 108 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:23.527961 28501 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 108.101, Last appended by leader: 101, Current term: 108, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.528019 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 108 FOLLOWER]: Advancing to term 109
I20250113 21:10:23.598938 28511 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 108 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:23.599056 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 108 FOLLOWER]: Advancing to term 109
I20250113 21:10:23.599702 28511 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.599856 28511 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 109 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:23.600085 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 109 candidate_status { last_received { term: 108 index: 101 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:23.600085 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 109 candidate_status { last_received { term: 108 index: 101 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:23.600229 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 109 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 109: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:23.600731 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 109.
I20250113 21:10:23.600910 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:23.601001 28511 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 109 FOLLOWER]: Leader election won for term 109
I20250113 21:10:23.601049 28511 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 109 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:23.601089 28511 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 108.101, Last appended by leader: 101, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.601534 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 108 to 109, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 109 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:23.653309 28512 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 109 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:23.653411 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 109 FOLLOWER]: Advancing to term 110
I20250113 21:10:23.654098 28512 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.654261 28512 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 110 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:23.654460 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 110 candidate_status { last_received { term: 108 index: 101 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:23.654469 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 110 candidate_status { last_received { term: 108 index: 101 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:23.654551 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 109 FOLLOWER]: Advancing to term 110
I20250113 21:10:23.655086 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 110 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 110.
I20250113 21:10:23.655232 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 110 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:23.655318 28512 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 110 FOLLOWER]: Leader election won for term 110
I20250113 21:10:23.655388 28512 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 110 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:23.655426 28512 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 108.101, Last appended by leader: 101, Current term: 110, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.655823 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 109 to 110, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 110 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:23.695276 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 110 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 109. Current term is 110. Ops: []
I20250113 21:10:23.695526 28511 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250113 21:10:23.695658 28511 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 109 LEADER]: Stepping down as leader of term 109
I20250113 21:10:23.695696 28511 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 109 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:23.695760 28511 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 109.102, Last appended by leader: 102, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.695824 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 109 FOLLOWER]: Advancing to term 110
I20250113 21:10:23.732801 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:23.756394 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 110 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 109 index: 102. Preceding OpId from leader: term: 110 index: 102. (term mismatch)
I20250113 21:10:23.756471 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 101
I20250113 21:10:23.756498 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 109.102
I20250113 21:10:23.756521 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 110 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:23.756832 28512 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250113 21:10:23.758056 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 110 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 108 index: 101. Preceding OpId from leader: term: 110 index: 102. (index mismatch)
I20250113 21:10:23.758235 28516 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250113 21:10:23.944172 28511 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 110 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:23.944274 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 110 FOLLOWER]: Advancing to term 111
I20250113 21:10:23.944923 28511 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.945067 28511 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 111 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:23.945317 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 111 candidate_status { last_received { term: 110 index: 102 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:23.945365 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 111 candidate_status { last_received { term: 110 index: 102 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:23.945590 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 111 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:23.945679 28511 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 111 FOLLOWER]: Leader election lost for term 111. Reason: could not achieve majority
I20250113 21:10:23.961483 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 111 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 110. Current term is 111. Ops: []
I20250113 21:10:23.961733 28514 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 110.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250113 21:10:23.961894 28512 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 110 LEADER]: Stepping down as leader of term 110
I20250113 21:10:23.961935 28512 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 110 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:23.961990 28512 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 110.102, Last appended by leader: 102, Current term: 110, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:23.962064 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 110 FOLLOWER]: Advancing to term 111
I20250113 21:10:24.022269 28501 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 110 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:24.022364 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 110 FOLLOWER]: Advancing to term 111
I20250113 21:10:24.023027 28501 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.023191 28501 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 111 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:24.023398 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 111 candidate_status { last_received { term: 110 index: 102 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:24.023403 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 111 candidate_status { last_received { term: 110 index: 102 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:24.023525 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 111 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 111: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:24.024034 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 111.
I20250113 21:10:24.024220 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:24.024294 28501 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 111 FOLLOWER]: Leader election won for term 111
I20250113 21:10:24.024434 28501 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 111 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:24.024541 28501 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 110.102, Last appended by leader: 102, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.025027 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 110 to 111, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 111 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:24.075963 28511 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 111 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:24.076071 28511 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 111 FOLLOWER]: Advancing to term 112
I20250113 21:10:24.076824 28511 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.077000 28511 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 112 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:24.077217 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 112 candidate_status { last_received { term: 110 index: 102 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:24.077289 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 111 FOLLOWER]: Advancing to term 112
I20250113 21:10:24.077221 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 112 candidate_status { last_received { term: 110 index: 102 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:24.077945 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 112 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 112.
I20250113 21:10:24.078107 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 112 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:24.078189 28511 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 112 FOLLOWER]: Leader election won for term 112
I20250113 21:10:24.078243 28511 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 112 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:24.078284 28511 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 110.102, Last appended by leader: 102, Current term: 112, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.078719 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 111 to 112, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 112 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:24.117547 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 112 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 111. Current term is 112. Ops: []
I20250113 21:10:24.117887 28501 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250113 21:10:24.118003 28501 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 112.103 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 102
I20250113 21:10:24.118052 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 111 LEADER]: Stepping down as leader of term 111
I20250113 21:10:24.118093 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 111 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:24.118173 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 111.103, Last appended by leader: 103, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.118259 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 111 FOLLOWER]: Advancing to term 112
I20250113 21:10:24.159077 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 112 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 111 index: 103. Preceding OpId from leader: term: 112 index: 103. (term mismatch)
I20250113 21:10:24.159155 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 102
I20250113 21:10:24.159176 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 111.103
I20250113 21:10:24.159198 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 112 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:24.159440 28511 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250113 21:10:24.160423 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 112 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 110 index: 102. Preceding OpId from leader: term: 112 index: 103. (index mismatch)
I20250113 21:10:24.160615 28511 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250113 21:10:24.436328 28512 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 112 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:24.436430 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 112 FOLLOWER]: Advancing to term 113
I20250113 21:10:24.437078 28512 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.437247 28512 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 113 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:24.437484 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 113 candidate_status { last_received { term: 112 index: 103 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:24.437484 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 113 candidate_status { last_received { term: 112 index: 103 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:24.437801 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 113 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:24.437894 28512 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 113 FOLLOWER]: Leader election lost for term 113. Reason: could not achieve majority
I20250113 21:10:24.439121 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 113 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 112. Current term is 113. Ops: []
I20250113 21:10:24.439320 28517 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 112.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250113 21:10:24.439425 28517 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 112 LEADER]: Stepping down as leader of term 112
I20250113 21:10:24.439460 28517 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 112 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:24.439515 28517 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 112.103, Last appended by leader: 103, Current term: 112, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.439599 28517 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 112 FOLLOWER]: Advancing to term 113
I20250113 21:10:24.461974 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 112 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:24.462092 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 112 FOLLOWER]: Advancing to term 113
I20250113 21:10:24.462752 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.462940 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 113 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:24.463135 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 113 candidate_status { last_received { term: 112 index: 103 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:24.463116 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 113 candidate_status { last_received { term: 112 index: 103 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:24.463215 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 113 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 113: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:24.463843 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 113.
I20250113 21:10:24.464010 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:24.464089 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 113 FOLLOWER]: Leader election won for term 113
I20250113 21:10:24.464144 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 113 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:24.464197 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 112.103, Last appended by leader: 103, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.464628 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 112 to 113, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 113 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:24.545485 28512 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 113 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:24.545588 28512 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 113 FOLLOWER]: Advancing to term 114
I20250113 21:10:24.546257 28512 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.546437 28512 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 114 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:24.546658 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 114 candidate_status { last_received { term: 112 index: 103 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:24.546667 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 114 candidate_status { last_received { term: 112 index: 103 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:24.546770 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 113 FOLLOWER]: Advancing to term 114
I20250113 21:10:24.547297 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 114 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 114.
I20250113 21:10:24.547461 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 114 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:24.547555 28512 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 114 FOLLOWER]: Leader election won for term 114
I20250113 21:10:24.547657 28512 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 114 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:24.547698 28512 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 112.103, Last appended by leader: 103, Current term: 114, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.548125 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 113 to 114, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 114 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:24.561933 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 114 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 113. Current term is 114. Ops: []
I20250113 21:10:24.562160 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250113 21:10:24.562283 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 113 LEADER]: Stepping down as leader of term 113
I20250113 21:10:24.562315 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 113 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:24.562377 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.104, Last appended by leader: 104, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.562443 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 113 FOLLOWER]: Advancing to term 114
I20250113 21:10:24.627521 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 114 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 113 index: 104. Preceding OpId from leader: term: 114 index: 104. (term mismatch)
I20250113 21:10:24.627604 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 103
I20250113 21:10:24.627626 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 113.104
I20250113 21:10:24.627651 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 114 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:24.627882 28512 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250113 21:10:24.628842 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 114 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 112 index: 103. Preceding OpId from leader: term: 114 index: 104. (index mismatch)
I20250113 21:10:24.629091 28518 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250113 21:10:24.733063 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:24.808401 28517 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 114 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:24.808503 28517 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 114 FOLLOWER]: Advancing to term 115
I20250113 21:10:24.809219 28517 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.809386 28517 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 115 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:24.809594 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:24.809594 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:24.809985 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 115 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:24.810081 28517 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 115 FOLLOWER]: Leader election lost for term 115. Reason: could not achieve majority
I20250113 21:10:24.812958 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 115 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 114. Current term is 115. Ops: []
I20250113 21:10:24.813164 28519 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 114.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250113 21:10:24.813253 28519 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 114 LEADER]: Stepping down as leader of term 114
I20250113 21:10:24.813285 28519 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 114 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:24.813346 28519 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 114.104, Last appended by leader: 104, Current term: 114, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.813416 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 114 FOLLOWER]: Advancing to term 115
I20250113 21:10:24.831132 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 114 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:24.831228 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 114 FOLLOWER]: Advancing to term 115
I20250113 21:10:24.831851 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.832000 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 115 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:24.832221 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:24.832221 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:24.832324 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 115: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:24.832861 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 115.
I20250113 21:10:24.833039 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:24.833115 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 115 FOLLOWER]: Leader election won for term 115
I20250113 21:10:24.833165 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 115 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:24.833223 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 114.104, Last appended by leader: 104, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.833676 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 114 to 115, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 115 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:24.928458 28517 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 115 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:24.928558 28517 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 115 FOLLOWER]: Advancing to term 116
I20250113 21:10:24.929242 28517 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.929428 28517 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 116 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:24.929661 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 116 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:24.929661 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 116 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:24.929759 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 115 FOLLOWER]: Advancing to term 116
I20250113 21:10:24.930338 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 116 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 116.
I20250113 21:10:24.930506 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 116 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:24.930622 28517 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 116 FOLLOWER]: Leader election won for term 116
I20250113 21:10:24.930682 28517 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 116 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:24.930738 28517 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 114.104, Last appended by leader: 104, Current term: 116, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.931174 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 115 to 116, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 116 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:24.932180 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 116 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 115. Current term is 116. Ops: []
I20250113 21:10:24.932390 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250113 21:10:24.932471 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 115 LEADER]: Stepping down as leader of term 115
I20250113 21:10:24.932493 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 115 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:24.932536 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 115.105, Last appended by leader: 105, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:24.932615 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 115 FOLLOWER]: Advancing to term 116
I20250113 21:10:25.023783 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 116 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 114 index: 104. Preceding OpId from leader: term: 116 index: 105. (index mismatch)
I20250113 21:10:25.024107 28517 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250113 21:10:25.025182 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 116 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 115 index: 105. Preceding OpId from leader: term: 116 index: 105. (term mismatch)
I20250113 21:10:25.025240 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 104
I20250113 21:10:25.025264 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 115.105
I20250113 21:10:25.025288 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 116 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:25.025482 28520 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250113 21:10:25.208519 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 116 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:25.208643 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 116 FOLLOWER]: Advancing to term 117
I20250113 21:10:25.209308 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.209481 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 117 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:25.209695 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:25.209707 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:25.209951 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 117 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:25.210049 28519 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 117 FOLLOWER]: Leader election lost for term 117. Reason: could not achieve majority
I20250113 21:10:25.218379 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 117 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 116. Current term is 117. Ops: []
I20250113 21:10:25.218626 28520 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 116.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250113 21:10:25.218751 28517 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 116 LEADER]: Stepping down as leader of term 116
I20250113 21:10:25.218801 28517 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 116 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:25.218858 28517 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 116.105, Last appended by leader: 105, Current term: 116, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.218936 28517 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 116 FOLLOWER]: Advancing to term 117
I20250113 21:10:25.241789 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 116 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:25.241921 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 116 FOLLOWER]: Advancing to term 117
I20250113 21:10:25.242566 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.242722 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 117 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:25.242947 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:25.242955 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:25.243044 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 117 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 117: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:25.243592 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 117.
I20250113 21:10:25.243763 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:25.243852 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 117 FOLLOWER]: Leader election won for term 117
I20250113 21:10:25.243901 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 117 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:25.243935 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 116.105, Last appended by leader: 105, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.244379 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 116 to 117, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 117 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:25.310424 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 117 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:25.310513 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 117 FOLLOWER]: Advancing to term 118
I20250113 21:10:25.311182 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.311338 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 118 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:25.311574 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 118 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:25.311574 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 118 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:25.311679 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 117 FOLLOWER]: Advancing to term 118
I20250113 21:10:25.312232 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 118 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 118.
I20250113 21:10:25.312417 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 118 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:25.312495 28519 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 118 FOLLOWER]: Leader election won for term 118
I20250113 21:10:25.312556 28519 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 118 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:25.312618 28519 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 116.105, Last appended by leader: 105, Current term: 118, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.313086 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 117 to 118, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 118 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:25.357461 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 118 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 117. Current term is 118. Ops: []
I20250113 21:10:25.357715 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250113 21:10:25.357806 28515 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 118.106 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 105
I20250113 21:10:25.357841 28501 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 117 LEADER]: Stepping down as leader of term 117
I20250113 21:10:25.357897 28501 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 117 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:25.357964 28501 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 117.106, Last appended by leader: 106, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.358032 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 117 FOLLOWER]: Advancing to term 118
I20250113 21:10:25.392622 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 118 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 116 index: 105. Preceding OpId from leader: term: 118 index: 106. (index mismatch)
I20250113 21:10:25.392923 28519 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250113 21:10:25.393838 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 118 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 117 index: 106. Preceding OpId from leader: term: 118 index: 106. (term mismatch)
I20250113 21:10:25.393911 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 105
I20250113 21:10:25.393931 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 117.106
I20250113 21:10:25.393949 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 118 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:25.394146 28521 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250113 21:10:25.494475 28517 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 118 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:25.494570 28517 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 118 FOLLOWER]: Advancing to term 119
I20250113 21:10:25.495213 28517 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.495401 28517 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 119 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:25.495553 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 119 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:25.495573 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 119 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:25.495770 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 119 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:25.495865 28517 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 119 FOLLOWER]: Leader election lost for term 119. Reason: could not achieve majority
I20250113 21:10:25.496788 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 119 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 118. Current term is 119. Ops: []
I20250113 21:10:25.496984 28519 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 118.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250113 21:10:25.497074 28519 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 118 LEADER]: Stepping down as leader of term 118
I20250113 21:10:25.497107 28519 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 118 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:25.497164 28519 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 118.106, Last appended by leader: 106, Current term: 118, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.497231 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 118 FOLLOWER]: Advancing to term 119
I20250113 21:10:25.591181 28501 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 118 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:25.591286 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 118 FOLLOWER]: Advancing to term 119
I20250113 21:10:25.591964 28501 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.592157 28501 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 119 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:25.592378 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 119 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:25.592396 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 119 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:25.592491 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 119 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 119: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:25.593000 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 119.
I20250113 21:10:25.593173 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 119 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:25.593264 28501 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 119 FOLLOWER]: Leader election won for term 119
I20250113 21:10:25.593312 28501 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 119 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:25.593358 28501 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 118.106, Last appended by leader: 106, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.593807 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 118 to 119, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 119 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:25.645336 28517 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 119 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:25.645442 28517 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 119 FOLLOWER]: Advancing to term 120
I20250113 21:10:25.646128 28517 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.646276 28517 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 120 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:25.646500 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 120 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:25.646500 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 120 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:25.646623 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 119 FOLLOWER]: Advancing to term 120
I20250113 21:10:25.647158 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 120.
I20250113 21:10:25.647333 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 120 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:25.647430 28517 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 120 FOLLOWER]: Leader election won for term 120
I20250113 21:10:25.647491 28517 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 120 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:25.647547 28517 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 118.106, Last appended by leader: 106, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.647961 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 119 to 120, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 120 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:25.693305 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 120 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 119. Current term is 120. Ops: []
I20250113 21:10:25.693617 28501 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250113 21:10:25.693704 28501 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 59ff47de1d4340e8b03823c58cb98738 log is divergent from this leader: its last log entry 120.107 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 106
I20250113 21:10:25.693753 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 119 LEADER]: Stepping down as leader of term 119
I20250113 21:10:25.693794 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 119 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:25.693843 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 119.107, Last appended by leader: 107, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:25.693948 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 119 FOLLOWER]: Advancing to term 120
I20250113 21:10:25.733321 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:25.752512 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 120 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 118 index: 106. Preceding OpId from leader: term: 120 index: 107. (index mismatch)
I20250113 21:10:25.752790 28517 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250113 21:10:25.753746 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 120 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 119 index: 107. Preceding OpId from leader: term: 120 index: 107. (term mismatch)
I20250113 21:10:25.753800 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 106
I20250113 21:10:25.753824 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 119.107
I20250113 21:10:25.753850 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 120 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:25.754065 28520 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250113 21:10:26.043699 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 120 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:26.043812 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 120 FOLLOWER]: Advancing to term 121
I20250113 21:10:26.044486 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.044656 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 121 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:26.044858 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 121 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:26.044878 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 121 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:26.045111 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 121 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:26.045199 28515 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 121 FOLLOWER]: Leader election lost for term 121. Reason: could not achieve majority
I20250113 21:10:26.065593 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 121 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 120. Current term is 121. Ops: []
I20250113 21:10:26.065891 28517 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 120.107, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250113 21:10:26.066030 28520 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 120 LEADER]: Stepping down as leader of term 120
I20250113 21:10:26.066095 28520 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 120 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:26.066176 28520 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 120.107, Last appended by leader: 107, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.066264 28520 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 120 FOLLOWER]: Advancing to term 121
I20250113 21:10:26.132862 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 120 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:26.132987 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 120 FOLLOWER]: Advancing to term 121
I20250113 21:10:26.133664 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.133807 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 121 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:26.134056 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 121 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:26.134063 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 121 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:26.134182 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 121 FOLLOWER]: Leader election vote request: Denying vote to candidate 76a522394a8041a18ebb4e478f89c29f in current term 121: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:26.134721 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 121.
I20250113 21:10:26.134902 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 121 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:26.134994 28519 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 121 FOLLOWER]: Leader election won for term 121
I20250113 21:10:26.135093 28519 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 121 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:26.135149 28519 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 120.107, Last appended by leader: 107, Current term: 121, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.135574 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 120 to 121, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 121 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:26.180076 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 121 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:26.180176 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 121 FOLLOWER]: Advancing to term 122
I20250113 21:10:26.180807 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.180953 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 122 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:26.181188 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 122 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:26.181174 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 122 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:26.181293 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 121 FOLLOWER]: Advancing to term 122
I20250113 21:10:26.181895 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 122.
I20250113 21:10:26.182065 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:26.182163 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 122 FOLLOWER]: Leader election won for term 122
I20250113 21:10:26.182220 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 122 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:26.182293 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 120.107, Last appended by leader: 107, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.182798 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 121 to 122, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 122 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:26.247016 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 122 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 121. Current term is 122. Ops: []
I20250113 21:10:26.247321 28519 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250113 21:10:26.247444 28519 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 121 LEADER]: Stepping down as leader of term 121
I20250113 21:10:26.247485 28519 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 121 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:26.247534 28519 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.108, Last appended by leader: 108, Current term: 121, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.247586 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 121 FOLLOWER]: Advancing to term 122
I20250113 21:10:26.290308 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 122 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 121 index: 108. Preceding OpId from leader: term: 122 index: 108. (term mismatch)
I20250113 21:10:26.290393 28256 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting all ops after (but not including) 107
I20250113 21:10:26.290413 28256 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting uncommitted NO_OP operation due to leader change: 121.108
I20250113 21:10:26.290434 28256 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 122 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:26.290657 28515 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250113 21:10:26.291596 28520 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 122 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:26.291788 28520 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 122 FOLLOWER]: Advancing to term 123
I20250113 21:10:26.292495 28520 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.292606 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 123 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 122. Current term is 123. Ops: []
I20250113 21:10:26.292726 28520 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 123 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:26.292757 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 123 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:26.292871 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 123 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:26.293020 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250113 21:10:26.293057 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:26.293107 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 122 LEADER]: Stepping down as leader of term 122
I20250113 21:10:26.293140 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 122 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:26.293140 28520 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
I20250113 21:10:26.293277 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 122.108, Last appended by leader: 108, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.293344 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 122 FOLLOWER]: Advancing to term 123
I20250113 21:10:26.406632 28520 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 123 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:26.406725 28520 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 123 FOLLOWER]: Advancing to term 124
I20250113 21:10:26.407378 28520 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.407543 28520 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 124 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:26.407740 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 124 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:26.407740 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 124 candidate_status { last_received { term: 120 index: 107 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:26.407838 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 123 FOLLOWER]: Advancing to term 124
I20250113 21:10:26.407847 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 122 FOLLOWER]: Advancing to term 124
I20250113 21:10:26.408494 28179 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 124 because replica has last-logged OpId of term: 122 index: 108, which is greater than that of the candidate, which has last-logged OpId of term: 120 index: 107.
I20250113 21:10:26.408494 28256 raft_consensus.cc:2405] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 for term 124 because replica has last-logged OpId of term: 122 index: 108, which is greater than that of the candidate, which has last-logged OpId of term: 120 index: 107.
I20250113 21:10:26.408739 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 124 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:26.408841 28520 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 124 FOLLOWER]: Leader election lost for term 124. Reason: could not achieve majority
I20250113 21:10:26.473016 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 124 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:26.473124 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 124 FOLLOWER]: Advancing to term 125
I20250113 21:10:26.473784 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.473982 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 125 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:26.474200 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 125 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:26.474283 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 124 FOLLOWER]: Advancing to term 125
I20250113 21:10:26.474217 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 125 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:26.474328 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 124 FOLLOWER]: Advancing to term 125
I20250113 21:10:26.474825 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 125.
I20250113 21:10:26.474848 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 125.
I20250113 21:10:26.475019 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 125 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:26.475128 28519 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 125 FOLLOWER]: Leader election won for term 125
I20250113 21:10:26.475190 28519 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 125 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:26.475244 28519 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 122.108, Last appended by leader: 108, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.475715 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 122 to 125, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 125 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:26.560093 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 125 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 120 index: 107. Preceding OpId from leader: term: 125 index: 109. (index mismatch)
I20250113 21:10:26.560353 28519 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 107, Time since last communication: 0.000s
I20250113 21:10:26.561317 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 125 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 122 index: 108. Preceding OpId from leader: term: 125 index: 109. (index mismatch)
I20250113 21:10:26.561504 28523 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250113 21:10:26.661630 28520 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 125 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:26.661722 28520 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 125 FOLLOWER]: Advancing to term 126
I20250113 21:10:26.662408 28520 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.662565 28520 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 126 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:26.662745 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:26.662748 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:26.662983 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 126 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:26.663076 28520 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 126 FOLLOWER]: Leader election lost for term 126. Reason: could not achieve majority
I20250113 21:10:26.685250 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 126 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 125. Current term is 126. Ops: []
I20250113 21:10:26.685508 28519 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 125.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250113 21:10:26.685632 28519 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 125 LEADER]: Stepping down as leader of term 125
I20250113 21:10:26.685665 28519 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 125 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:26.685719 28519 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.109, Last appended by leader: 109, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.685788 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 125 FOLLOWER]: Advancing to term 126
I20250113 21:10:26.733567 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:26.750466 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 125 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:26.750581 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 125 FOLLOWER]: Advancing to term 126
I20250113 21:10:26.751219 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.751382 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 126 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:26.751624 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:26.751720 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 126 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 126: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:26.751622 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:26.752316 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 126.
I20250113 21:10:26.752502 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:26.752605 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 126 FOLLOWER]: Leader election won for term 126
I20250113 21:10:26.752671 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 126 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:26.752728 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.109, Last appended by leader: 109, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.753165 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 125 to 126, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 126 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:26.795519 28520 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 126 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:26.795612 28520 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 126 FOLLOWER]: Advancing to term 127
I20250113 21:10:26.796277 28520 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.796447 28520 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 127 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:26.796715 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 127 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:26.796823 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 126 FOLLOWER]: Advancing to term 127
I20250113 21:10:26.797670 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 127.
I20250113 21:10:26.797917 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 127 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters:
I20250113 21:10:26.797919 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 127 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:26.798027 28520 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 127 FOLLOWER]: Leader election won for term 127
I20250113 21:10:26.798123 28520 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 127 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:26.798242 28520 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.109, Last appended by leader: 109, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.798693 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 126 to 127, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 127 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:26.828491 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 127 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 126. Current term is 127. Ops: []
I20250113 21:10:26.828766 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250113 21:10:26.828874 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 126 LEADER]: Stepping down as leader of term 126
I20250113 21:10:26.828910 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 126 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:26.828967 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 126.110, Last appended by leader: 110, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.829018 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 126 FOLLOWER]: Advancing to term 127
I20250113 21:10:26.885172 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 127 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 125 index: 109. Preceding OpId from leader: term: 127 index: 110. (index mismatch)
I20250113 21:10:26.885476 28520 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250113 21:10:26.886446 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 127 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 126 index: 110. Preceding OpId from leader: term: 127 index: 110. (term mismatch)
I20250113 21:10:26.886493 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 109
I20250113 21:10:26.886514 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 126.110
I20250113 21:10:26.886539 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 127 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:26.886719 28524 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250113 21:10:26.986964 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 127 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:26.987057 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 127 FOLLOWER]: Advancing to term 128
I20250113 21:10:26.987717 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:26.987879 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 128 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:26.988063 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:26.988067 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:26.988317 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 128 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:26.988399 28519 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 128 FOLLOWER]: Leader election lost for term 128. Reason: could not achieve majority
I20250113 21:10:27.010660 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 128 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 127. Current term is 128. Ops: []
I20250113 21:10:27.010970 28525 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 127.110, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250113 21:10:27.011108 28525 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 127 LEADER]: Stepping down as leader of term 127
I20250113 21:10:27.011147 28525 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 127 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:27.011201 28525 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 127.110, Last appended by leader: 110, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.011287 28525 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 127 FOLLOWER]: Advancing to term 128
I20250113 21:10:27.078569 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 127 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:27.078672 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 127 FOLLOWER]: Advancing to term 128
I20250113 21:10:27.079347 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.079506 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 128 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:27.079727 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:27.079746 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:27.079830 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 128 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 128: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:27.080345 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 128 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 128.
I20250113 21:10:27.080559 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 128 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:27.080637 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 128 FOLLOWER]: Leader election won for term 128
I20250113 21:10:27.080686 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 128 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:27.080742 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 127.110, Last appended by leader: 110, Current term: 128, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.081199 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 127 to 128, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 128 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:27.126230 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 128 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:27.126330 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 128 FOLLOWER]: Advancing to term 129
I20250113 21:10:27.126931 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.127105 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 129 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:27.127337 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 129 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:27.127408 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 128 FOLLOWER]: Advancing to term 129
I20250113 21:10:27.127341 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 129 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:27.127940 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 129.
I20250113 21:10:27.128110 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:27.128209 28519 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 129 FOLLOWER]: Leader election won for term 129
I20250113 21:10:27.128309 28519 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 129 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:27.128363 28519 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 127.110, Last appended by leader: 110, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.128813 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 128 to 129, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 129 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:27.169745 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 129 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 128. Current term is 129. Ops: []
I20250113 21:10:27.170042 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250113 21:10:27.170162 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 128 LEADER]: Stepping down as leader of term 128
I20250113 21:10:27.170202 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 128 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:27.170267 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 128.111, Last appended by leader: 111, Current term: 128, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.170354 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 128 FOLLOWER]: Advancing to term 129
I20250113 21:10:27.221187 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 129 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 128 index: 111. Preceding OpId from leader: term: 129 index: 111. (term mismatch)
I20250113 21:10:27.221263 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 110
I20250113 21:10:27.221277 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 128.111
I20250113 21:10:27.221292 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 129 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:27.221529 28519 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250113 21:10:27.222568 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 129 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 127 index: 110. Preceding OpId from leader: term: 129 index: 111. (index mismatch)
I20250113 21:10:27.222787 28519 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250113 21:10:27.412247 28525 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 129 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:27.412338 28525 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 129 FOLLOWER]: Advancing to term 130
I20250113 21:10:27.413280 28525 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.413512 28525 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 130 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:27.413720 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:27.413720 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:27.414022 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 130 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:27.414124 28525 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 130 FOLLOWER]: Leader election lost for term 130. Reason: could not achieve majority
I20250113 21:10:27.425570 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 130 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 129. Current term is 130. Ops: []
I20250113 21:10:27.425815 28526 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 129.111, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250113 21:10:27.425971 28519 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 129 LEADER]: Stepping down as leader of term 129
I20250113 21:10:27.426012 28519 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 129 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:27.426066 28519 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.111, Last appended by leader: 111, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.426146 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 129 FOLLOWER]: Advancing to term 130
I20250113 21:10:27.495072 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 129 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:27.495169 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 129 FOLLOWER]: Advancing to term 130
I20250113 21:10:27.495829 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.495993 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 130 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:27.496222 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:27.496289 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 130 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 130: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:27.496241 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:27.496904 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 130.
I20250113 21:10:27.497087 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 130 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:27.497195 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 130 FOLLOWER]: Leader election won for term 130
I20250113 21:10:27.497298 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 130 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:27.497359 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.111, Last appended by leader: 111, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.497804 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 129 to 130, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 130 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:27.553608 28525 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 130 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:27.553701 28525 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 130 FOLLOWER]: Advancing to term 131
I20250113 21:10:27.554437 28525 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.554595 28525 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 131 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:27.554811 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 131 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:27.554811 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 131 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:27.554906 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 130 FOLLOWER]: Advancing to term 131
I20250113 21:10:27.555485 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 131.
I20250113 21:10:27.555641 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 131 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:27.555711 28525 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 131 FOLLOWER]: Leader election won for term 131
I20250113 21:10:27.555816 28525 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 131 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:27.555855 28525 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.111, Last appended by leader: 111, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.556305 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 130 to 131, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 131 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:27.587221 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 131 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 130. Current term is 131. Ops: []
I20250113 21:10:27.587495 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250113 21:10:27.587607 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 130 LEADER]: Stepping down as leader of term 130
I20250113 21:10:27.587644 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 130 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:27.587689 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 130.112, Last appended by leader: 112, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.587741 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 130 FOLLOWER]: Advancing to term 131
I20250113 21:10:27.645887 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 131 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 129 index: 111. Preceding OpId from leader: term: 131 index: 112. (index mismatch)
I20250113 21:10:27.646178 28525 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250113 21:10:27.647183 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 131 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 130 index: 112. Preceding OpId from leader: term: 131 index: 112. (term mismatch)
I20250113 21:10:27.647238 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 111
I20250113 21:10:27.647261 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 130.112
I20250113 21:10:27.647293 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 131 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:27.647552 28528 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250113 21:10:27.733816 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:27.748044 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 131 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:27.748138 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 131 FOLLOWER]: Advancing to term 132
I20250113 21:10:27.748874 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.749059 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 132 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:27.749181 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:27.749222 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 132 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 131. Current term is 132. Ops: []
I20250113 21:10:27.749236 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:27.749459 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 132 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:27.749482 28528 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 131.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250113 21:10:27.749545 28519 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 132 FOLLOWER]: Leader election lost for term 132. Reason: could not achieve majority
I20250113 21:10:27.749598 28525 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 131 LEADER]: Stepping down as leader of term 131
I20250113 21:10:27.749634 28525 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 131 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:27.749681 28525 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 131.112, Last appended by leader: 112, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.749737 28525 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 131 FOLLOWER]: Advancing to term 132
I20250113 21:10:27.849547 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 131 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:27.849653 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 131 FOLLOWER]: Advancing to term 132
I20250113 21:10:27.850332 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.850495 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 132 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:27.850736 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:27.850819 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 132 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 132: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:27.850741 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:27.851404 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 132.
I20250113 21:10:27.851598 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:27.851680 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 132 FOLLOWER]: Leader election won for term 132
I20250113 21:10:27.851739 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 132 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:27.851792 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 131.112, Last appended by leader: 112, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.852254 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 131 to 132, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 132 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:27.892068 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 132 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:27.892169 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 132 FOLLOWER]: Advancing to term 133
I20250113 21:10:27.892807 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.892956 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 133 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:27.893159 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 133 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:27.893159 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 133 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:27.893255 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 132 FOLLOWER]: Advancing to term 133
I20250113 21:10:27.893779 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 133 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 133.
I20250113 21:10:27.893939 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 133 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:27.894037 28519 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 133 FOLLOWER]: Leader election won for term 133
I20250113 21:10:27.894110 28519 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 133 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:27.894152 28519 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 131.112, Last appended by leader: 112, Current term: 133, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.894596 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 132 to 133, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 133 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:27.959658 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 133 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 132. Current term is 133. Ops: []
I20250113 21:10:27.959934 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250113 21:10:27.960050 28515 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 133.113 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 112
I20250113 21:10:27.960098 28501 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 132 LEADER]: Stepping down as leader of term 132
I20250113 21:10:27.960170 28501 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 132 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:27.960222 28501 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 132.113, Last appended by leader: 113, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:27.960314 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 132 FOLLOWER]: Advancing to term 133
I20250113 21:10:27.983022 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 133 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 132 index: 113. Preceding OpId from leader: term: 133 index: 113. (term mismatch)
I20250113 21:10:27.983099 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 112
I20250113 21:10:27.983112 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 132.113
I20250113 21:10:27.983127 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 133 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:27.983354 28519 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250113 21:10:27.984350 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 133 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 131 index: 112. Preceding OpId from leader: term: 133 index: 113. (index mismatch)
I20250113 21:10:27.984576 28526 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250113 21:10:28.085346 28525 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 133 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:28.085438 28525 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 133 FOLLOWER]: Advancing to term 134
I20250113 21:10:28.086134 28525 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.086306 28525 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 134 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:28.086521 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 134 candidate_status { last_received { term: 133 index: 113 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:28.086534 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 134 candidate_status { last_received { term: 133 index: 113 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:28.086652 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 133 FOLLOWER]: Advancing to term 134
I20250113 21:10:28.087200 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 134.
I20250113 21:10:28.087363 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 134 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:28.087443 28525 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 134 FOLLOWER]: Leader election won for term 134
I20250113 21:10:28.087509 28525 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 134 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:28.087566 28525 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 133.113, Last appended by leader: 113, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.087962 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 133 to 134, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 134 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:28.098137 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 134 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 133. Current term is 134. Ops: []
I20250113 21:10:28.098322 28526 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 133.113, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250113 21:10:28.098418 28526 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 133 LEADER]: Stepping down as leader of term 133
I20250113 21:10:28.098453 28526 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 133 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:28.098524 28526 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 133.113, Last appended by leader: 113, Current term: 133, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.098603 28526 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 133 FOLLOWER]: Advancing to term 134
I20250113 21:10:28.187259 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 134 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 133 index: 113. Preceding OpId from leader: term: 134 index: 114. (index mismatch)
I20250113 21:10:28.187539 28525 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250113 21:10:28.188504 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 134 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 133 index: 113. Preceding OpId from leader: term: 134 index: 114. (index mismatch)
I20250113 21:10:28.188807 28525 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250113 21:10:28.290108 28501 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 134 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:28.290230 28501 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 134 FOLLOWER]: Advancing to term 135
I20250113 21:10:28.290896 28501 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.291054 28501 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 135 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:28.291275 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 135 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:28.291277 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 135 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:28.291407 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 134 FOLLOWER]: Advancing to term 135
I20250113 21:10:28.291925 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 135 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 135.
I20250113 21:10:28.292085 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 135 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:28.292187 28501 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 135 FOLLOWER]: Leader election won for term 135
I20250113 21:10:28.292251 28501 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 135 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:28.292325 28501 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 134.114, Last appended by leader: 114, Current term: 135, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.292769 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 134 to 135, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 135 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:28.301425 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 135 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 134. Current term is 135. Ops: []
I20250113 21:10:28.301658 28525 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 134.114, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250113 21:10:28.301755 28525 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 134 LEADER]: Stepping down as leader of term 134
I20250113 21:10:28.301779 28525 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 134 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:28.301837 28525 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 134.114, Last appended by leader: 114, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.301935 28525 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 134 FOLLOWER]: Advancing to term 135
I20250113 21:10:28.398272 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 135 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 134 index: 114. Preceding OpId from leader: term: 135 index: 115. (index mismatch)
I20250113 21:10:28.398563 28501 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250113 21:10:28.399605 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 135 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 134 index: 114. Preceding OpId from leader: term: 135 index: 115. (index mismatch)
I20250113 21:10:28.399767 28515 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250113 21:10:28.500841 28526 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 135 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:28.500931 28526 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 135 FOLLOWER]: Advancing to term 136
I20250113 21:10:28.501582 28526 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.501737 28526 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 136 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:28.501901 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 136 candidate_status { last_received { term: 135 index: 115 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:28.501925 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 136 candidate_status { last_received { term: 135 index: 115 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:28.502136 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 136 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:28.502228 28526 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 136 FOLLOWER]: Leader election lost for term 136. Reason: could not achieve majority
I20250113 21:10:28.511759 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 136 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 135. Current term is 136. Ops: []
I20250113 21:10:28.512001 28501 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 135.115, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250113 21:10:28.512135 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 135 LEADER]: Stepping down as leader of term 135
I20250113 21:10:28.512178 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 135 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:28.512244 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 135.115, Last appended by leader: 115, Current term: 135, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.512313 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 135 FOLLOWER]: Advancing to term 136
I20250113 21:10:28.600256 28525 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 135 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:28.600354 28525 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 135 FOLLOWER]: Advancing to term 136
I20250113 21:10:28.601044 28525 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.601267 28525 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 136 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:28.601500 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 136 candidate_status { last_received { term: 135 index: 115 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:28.601514 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 136 candidate_status { last_received { term: 135 index: 115 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:28.601615 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 136 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 136: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:28.602178 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 136.
I20250113 21:10:28.602357 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:28.602438 28525 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 136 FOLLOWER]: Leader election won for term 136
I20250113 21:10:28.602485 28525 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 136 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:28.602537 28525 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 135.115, Last appended by leader: 115, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.602995 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 135 to 136, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 136 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:28.652436 28526 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 136 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:28.652544 28526 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 136 FOLLOWER]: Advancing to term 137
I20250113 21:10:28.653285 28526 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.653473 28526 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 137 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:28.653712 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 137 candidate_status { last_received { term: 135 index: 115 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:28.653723 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 137 candidate_status { last_received { term: 135 index: 115 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:28.653822 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 136 FOLLOWER]: Advancing to term 137
I20250113 21:10:28.654424 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 137 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 137.
I20250113 21:10:28.654613 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 137 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:28.654721 28526 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 137 FOLLOWER]: Leader election won for term 137
I20250113 21:10:28.654791 28526 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 137 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:28.654856 28526 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 135.115, Last appended by leader: 115, Current term: 137, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.655309 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 136 to 137, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 137 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:28.700685 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 137 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 136. Current term is 137. Ops: []
I20250113 21:10:28.700946 28525 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250113 21:10:28.701052 28525 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 137.116 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 115
I20250113 21:10:28.701097 28528 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 136 LEADER]: Stepping down as leader of term 136
I20250113 21:10:28.701153 28528 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 136 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:28.701213 28528 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 136.116, Last appended by leader: 116, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.701350 28528 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 136 FOLLOWER]: Advancing to term 137
I20250113 21:10:28.734117 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:28.738736 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 137 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 135 index: 115. Preceding OpId from leader: term: 137 index: 116. (index mismatch)
I20250113 21:10:28.738991 28526 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250113 21:10:28.739904 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 137 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 136 index: 116. Preceding OpId from leader: term: 137 index: 116. (term mismatch)
I20250113 21:10:28.739952 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 115
I20250113 21:10:28.739974 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 136.116
I20250113 21:10:28.739997 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 137 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:28.740197 28526 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250113 21:10:28.840070 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 137 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:28.840166 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 137 FOLLOWER]: Advancing to term 138
I20250113 21:10:28.840860 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.841019 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 138 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:28.841207 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 138 candidate_status { last_received { term: 137 index: 116 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:28.841226 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 138 candidate_status { last_received { term: 137 index: 116 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:28.841279 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 137 FOLLOWER]: Advancing to term 138
I20250113 21:10:28.841818 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 138.
I20250113 21:10:28.842017 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:28.842103 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 138 FOLLOWER]: Leader election won for term 138
I20250113 21:10:28.842151 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 138 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:28.842190 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 137.116, Last appended by leader: 116, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.842742 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 137 to 138, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 138 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:28.845791 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 138 LEADER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 137. Current term is 138. Ops: []
I20250113 21:10:28.846010 28526 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 137.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250113 21:10:28.846089 28526 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 138.117 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 116
I20250113 21:10:28.846129 28519 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 137 LEADER]: Stepping down as leader of term 137
I20250113 21:10:28.846179 28519 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 137 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:28.846254 28519 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 137.116, Last appended by leader: 116, Current term: 137, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:28.846323 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 137 FOLLOWER]: Advancing to term 138
I20250113 21:10:28.929136 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 138 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 137 index: 116. Preceding OpId from leader: term: 138 index: 117. (index mismatch)
I20250113 21:10:28.929471 28515 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250113 21:10:28.930505 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 138 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 137 index: 116. Preceding OpId from leader: term: 138 index: 117. (index mismatch)
I20250113 21:10:28.930832 28530 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250113 21:10:29.253002 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 138 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:29.253100 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 138 FOLLOWER]: Advancing to term 139
I20250113 21:10:29.253731 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.253901 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 139 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:29.254092 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 139 candidate_status { last_received { term: 138 index: 117 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:29.254103 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 139 candidate_status { last_received { term: 138 index: 117 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:29.254333 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:29.254423 28519 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I20250113 21:10:29.258635 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 139 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 138. Current term is 139. Ops: []
I20250113 21:10:29.258828 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 138.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250113 21:10:29.258916 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 138 LEADER]: Stepping down as leader of term 138
I20250113 21:10:29.258932 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 138 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:29.258980 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 138.117, Last appended by leader: 117, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.259053 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 138 FOLLOWER]: Advancing to term 139
I20250113 21:10:29.307566 28528 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 138 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:29.307677 28528 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 138 FOLLOWER]: Advancing to term 139
I20250113 21:10:29.308398 28528 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.308583 28528 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 139 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:29.308776 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 139 candidate_status { last_received { term: 138 index: 117 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:29.308775 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 139 candidate_status { last_received { term: 138 index: 117 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:29.308871 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 139: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:29.309360 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 139.
I20250113 21:10:29.309547 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:29.309623 28528 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 139 FOLLOWER]: Leader election won for term 139
I20250113 21:10:29.309729 28528 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 139 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:29.309780 28528 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 138.117, Last appended by leader: 117, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.310227 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 138 to 139, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 139 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:29.386739 28519 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:29.386839 28519 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 139 FOLLOWER]: Advancing to term 140
I20250113 21:10:29.387524 28519 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.387693 28519 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 140 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:29.387902 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 140 candidate_status { last_received { term: 138 index: 117 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:29.387908 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 140 candidate_status { last_received { term: 138 index: 117 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:29.388034 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 139 FOLLOWER]: Advancing to term 140
I20250113 21:10:29.388473 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 140 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 139. Current term is 140. Ops: []
I20250113 21:10:29.388741 28528 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250113 21:10:29.388784 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 140 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 140.
I20250113 21:10:29.388832 28528 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 139 LEADER]: Stepping down as leader of term 139
I20250113 21:10:29.388868 28528 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 139 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:29.388914 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 140 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:29.388921 28528 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 139.118, Last appended by leader: 118, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.389017 28528 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 139 FOLLOWER]: Advancing to term 140
I20250113 21:10:29.389014 28519 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 140 FOLLOWER]: Leader election won for term 140
I20250113 21:10:29.389139 28519 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 140 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:29.389192 28519 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 138.117, Last appended by leader: 117, Current term: 140, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.389695 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 139 to 140, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 140 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:29.472044 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 140 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 138 index: 117. Preceding OpId from leader: term: 140 index: 118. (index mismatch)
I20250113 21:10:29.472338 28519 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250113 21:10:29.473253 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 140 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 139 index: 118. Preceding OpId from leader: term: 140 index: 118. (term mismatch)
I20250113 21:10:29.473316 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 117
I20250113 21:10:29.473336 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 139.118
I20250113 21:10:29.473366 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 140 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:29.473642 28519 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250113 21:10:29.661850 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 140 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:29.661965 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 140 FOLLOWER]: Advancing to term 141
I20250113 21:10:29.662606 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.662765 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 141 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:29.662986 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 141 candidate_status { last_received { term: 140 index: 118 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:29.662986 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 141 candidate_status { last_received { term: 140 index: 118 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:29.663206 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 141 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:29.663304 28515 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 141 FOLLOWER]: Leader election lost for term 141. Reason: could not achieve majority
I20250113 21:10:29.663385 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 141 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 140. Current term is 141. Ops: []
I20250113 21:10:29.663581 28533 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 140.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250113 21:10:29.663677 28533 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 140 LEADER]: Stepping down as leader of term 140
I20250113 21:10:29.663694 28533 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 140 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:29.663770 28533 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 140.118, Last appended by leader: 118, Current term: 140, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.663854 28533 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 140 FOLLOWER]: Advancing to term 141
I20250113 21:10:29.683959 28531 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 140 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:29.684033 28531 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 140 FOLLOWER]: Advancing to term 141
I20250113 21:10:29.684691 28531 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.684841 28531 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 141 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:29.685029 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 141 candidate_status { last_received { term: 140 index: 118 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:29.685029 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 141 candidate_status { last_received { term: 140 index: 118 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:29.685168 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 141 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 141: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:29.685696 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 141.
I20250113 21:10:29.685858 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 141 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:29.685949 28531 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 141 FOLLOWER]: Leader election won for term 141
I20250113 21:10:29.686030 28531 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 141 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:29.686089 28531 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 140.118, Last appended by leader: 118, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.686553 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 140 to 141, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 141 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:29.734366 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:29.766706 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 141 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:29.766793 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 141 FOLLOWER]: Advancing to term 142
I20250113 21:10:29.767465 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.767634 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 142 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:29.767864 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 142 candidate_status { last_received { term: 140 index: 118 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:29.767864 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 142 candidate_status { last_received { term: 140 index: 118 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:29.767971 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 141 FOLLOWER]: Advancing to term 142
I20250113 21:10:29.768533 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 142.
I20250113 21:10:29.768703 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:29.768808 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 142 FOLLOWER]: Leader election won for term 142
I20250113 21:10:29.768908 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 142 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:29.768949 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 140.118, Last appended by leader: 118, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.769435 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 141 to 142, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 142 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:29.781913 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 142 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 141. Current term is 142. Ops: []
I20250113 21:10:29.782151 28531 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250113 21:10:29.782264 28531 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 142.119 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 118
I20250113 21:10:29.782313 28528 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 141 LEADER]: Stepping down as leader of term 141
I20250113 21:10:29.782356 28528 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 141 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:29.782405 28528 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 141.119, Last appended by leader: 119, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.782509 28528 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 141 FOLLOWER]: Advancing to term 142
I20250113 21:10:29.882845 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 142 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 140 index: 118. Preceding OpId from leader: term: 142 index: 119. (index mismatch)
I20250113 21:10:29.883122 28515 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250113 21:10:29.884300 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 142 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 141 index: 119. Preceding OpId from leader: term: 142 index: 119. (term mismatch)
I20250113 21:10:29.884349 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 118
I20250113 21:10:29.884372 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 141.119
I20250113 21:10:29.884392 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 142 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:29.884555 28535 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250113 21:10:29.984535 28533 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 142 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:29.984625 28533 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 142 FOLLOWER]: Advancing to term 143
I20250113 21:10:29.985285 28533 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.985445 28533 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 143 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:29.985649 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 143 candidate_status { last_received { term: 142 index: 119 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:29.985654 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 143 candidate_status { last_received { term: 142 index: 119 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:29.985719 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 142 FOLLOWER]: Advancing to term 143
I20250113 21:10:29.986315 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 143 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 143.
I20250113 21:10:29.986488 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 143 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:29.986575 28533 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 143 FOLLOWER]: Leader election won for term 143
I20250113 21:10:29.986626 28533 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 143 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:29.986680 28533 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 142.119, Last appended by leader: 119, Current term: 143, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.987072 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 142 to 143, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 143 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:29.997795 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 143 LEADER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 142. Current term is 143. Ops: []
I20250113 21:10:29.997994 28535 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 142.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250113 21:10:29.998077 28535 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 143.120 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 119
I20250113 21:10:29.998118 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 142 LEADER]: Stepping down as leader of term 142
I20250113 21:10:29.998167 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 142 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:29.998217 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 142.119, Last appended by leader: 119, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:29.998283 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 142 FOLLOWER]: Advancing to term 143
I20250113 21:10:30.099469 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 143 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 142 index: 119. Preceding OpId from leader: term: 143 index: 120. (index mismatch)
I20250113 21:10:30.099794 28533 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250113 21:10:30.100718 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 143 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 142 index: 119. Preceding OpId from leader: term: 143 index: 120. (index mismatch)
I20250113 21:10:30.100879 28532 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250113 21:10:30.299122 28528 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 143 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:30.299196 28528 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 143 FOLLOWER]: Advancing to term 144
I20250113 21:10:30.299840 28528 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.300071 28528 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 144 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:30.300119 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 144 candidate_status { last_received { term: 143 index: 120 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:30.300258 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 144 candidate_status { last_received { term: 143 index: 120 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:30.300446 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 144 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:30.300537 28528 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 144 FOLLOWER]: Leader election lost for term 144. Reason: could not achieve majority
I20250113 21:10:30.323726 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 144 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 143. Current term is 144. Ops: []
I20250113 21:10:30.323973 28536 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 143.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250113 21:10:30.324087 28536 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 143 LEADER]: Stepping down as leader of term 143
I20250113 21:10:30.324115 28536 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 143 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:30.324172 28536 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 143.120, Last appended by leader: 120, Current term: 143, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.324268 28536 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 143 FOLLOWER]: Advancing to term 144
I20250113 21:10:30.395949 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 143 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:30.396044 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 143 FOLLOWER]: Advancing to term 144
I20250113 21:10:30.396683 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.396834 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 144 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:30.397066 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 144 candidate_status { last_received { term: 143 index: 120 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:30.397135 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 144 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 144: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:30.397081 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 144 candidate_status { last_received { term: 143 index: 120 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:30.398072 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 144.
I20250113 21:10:30.398263 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 144 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:30.398367 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 144 FOLLOWER]: Leader election won for term 144
I20250113 21:10:30.398448 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 144 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:30.398519 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 143.120, Last appended by leader: 120, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.399031 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 143 to 144, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 144 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:30.411958 28528 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 144 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:30.412015 28528 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 144 FOLLOWER]: Advancing to term 145
I20250113 21:10:30.412531 28528 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.412784 28528 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 145 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:30.412830 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 145 candidate_status { last_received { term: 143 index: 120 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:30.412920 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 144 FOLLOWER]: Advancing to term 145
I20250113 21:10:30.412957 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 145 candidate_status { last_received { term: 143 index: 120 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:30.413503 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 145 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 145.
I20250113 21:10:30.413666 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 145 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:30.413743 28528 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 145 FOLLOWER]: Leader election won for term 145
I20250113 21:10:30.413839 28528 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 145 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:30.413901 28528 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 143.120, Last appended by leader: 120, Current term: 145, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.414418 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 144 to 145, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 145 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:30.478438 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 145 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 144. Current term is 145. Ops: []
I20250113 21:10:30.478736 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250113 21:10:30.478852 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 144 LEADER]: Stepping down as leader of term 144
I20250113 21:10:30.478868 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 144 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:30.478942 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 144.121, Last appended by leader: 121, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.479015 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 144 FOLLOWER]: Advancing to term 145
I20250113 21:10:30.494701 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 145 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 144 index: 121. Preceding OpId from leader: term: 145 index: 121. (term mismatch)
I20250113 21:10:30.494764 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 120
I20250113 21:10:30.494777 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 144.121
I20250113 21:10:30.494792 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 145 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:30.495008 28528 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250113 21:10:30.495977 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 145 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 143 index: 120. Preceding OpId from leader: term: 145 index: 121. (index mismatch)
I20250113 21:10:30.496214 28538 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250113 21:10:30.597308 28536 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 145 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:30.597405 28536 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 145 FOLLOWER]: Advancing to term 146
I20250113 21:10:30.598135 28536 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.598307 28536 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 146 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:30.598536 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 146 candidate_status { last_received { term: 145 index: 121 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:30.598536 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 146 candidate_status { last_received { term: 145 index: 121 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:30.598645 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 145 FOLLOWER]: Advancing to term 146
I20250113 21:10:30.599167 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 146.
I20250113 21:10:30.599346 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 146 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:30.599431 28536 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 146 FOLLOWER]: Leader election won for term 146
I20250113 21:10:30.599488 28536 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 146 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:30.599547 28536 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 145.121, Last appended by leader: 121, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.599978 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 145 to 146, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 146 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:30.602531 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 146 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 145. Current term is 146. Ops: []
I20250113 21:10:30.602738 28538 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 145.121, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250113 21:10:30.602825 28538 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 145 LEADER]: Stepping down as leader of term 145
I20250113 21:10:30.602854 28538 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 145 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:30.602906 28538 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 145.121, Last appended by leader: 121, Current term: 145, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.602974 28538 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 145 FOLLOWER]: Advancing to term 146
I20250113 21:10:30.695506 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 146 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 145 index: 121. Preceding OpId from leader: term: 146 index: 122. (index mismatch)
I20250113 21:10:30.695782 28536 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250113 21:10:30.696846 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 146 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 145 index: 121. Preceding OpId from leader: term: 146 index: 122. (index mismatch)
I20250113 21:10:30.697062 28532 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250113 21:10:30.734826 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:30.887888 28535 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 146 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:30.887981 28535 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 146 FOLLOWER]: Advancing to term 147
I20250113 21:10:30.888705 28535 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.888864 28535 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 147 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:30.889068 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:30.889068 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:30.889295 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 147 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:30.889418 28535 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 147 FOLLOWER]: Leader election lost for term 147. Reason: could not achieve majority
I20250113 21:10:30.895505 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 147 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 146. Current term is 147. Ops: []
I20250113 21:10:30.895710 28536 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 146.122, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250113 21:10:30.895818 28536 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 146 LEADER]: Stepping down as leader of term 146
I20250113 21:10:30.895853 28536 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 146 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:30.895903 28536 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 146.122, Last appended by leader: 122, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.895977 28536 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 146 FOLLOWER]: Advancing to term 147
I20250113 21:10:30.920607 28538 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 146 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:30.920661 28538 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 146 FOLLOWER]: Advancing to term 147
I20250113 21:10:30.921187 28538 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.921320 28538 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 147 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:30.921492 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:30.921511 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:30.921558 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 147 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 147: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:30.922078 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 147.
I20250113 21:10:30.922258 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:30.922349 28538 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 147 FOLLOWER]: Leader election won for term 147
I20250113 21:10:30.922416 28538 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 147 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:30.922472 28538 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 146.122, Last appended by leader: 122, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:30.922886 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 146 to 147, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 147 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:31.029121 28535 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 147 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:31.029245 28535 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 147 FOLLOWER]: Advancing to term 148
I20250113 21:10:31.029994 28535 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.030165 28535 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 148 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:31.030376 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 148 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:31.030385 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 148 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:31.030514 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 147 FOLLOWER]: Advancing to term 148
I20250113 21:10:31.031059 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 148.
I20250113 21:10:31.031205 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:31.031309 28535 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 148 FOLLOWER]: Leader election won for term 148
I20250113 21:10:31.031375 28535 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 148 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:31.031431 28535 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 146.122, Last appended by leader: 122, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.031931 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 147 to 148, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 148 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:31.037076 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 148 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 147. Current term is 148. Ops: []
I20250113 21:10:31.037333 28538 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250113 21:10:31.037415 28538 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 147 LEADER]: Stepping down as leader of term 147
I20250113 21:10:31.037446 28538 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 147 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:31.037498 28538 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 147.123, Last appended by leader: 123, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.037562 28538 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 147 FOLLOWER]: Advancing to term 148
I20250113 21:10:31.119395 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 148 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 146 index: 122. Preceding OpId from leader: term: 148 index: 123. (index mismatch)
I20250113 21:10:31.119704 28535 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250113 21:10:31.120728 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 148 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 147 index: 123. Preceding OpId from leader: term: 148 index: 123. (term mismatch)
I20250113 21:10:31.120785 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 122
I20250113 21:10:31.120810 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 147.123
I20250113 21:10:31.120833 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 148 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:31.121037 28539 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250113 21:10:31.221653 28536 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 148 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:31.221758 28536 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 148 FOLLOWER]: Advancing to term 149
I20250113 21:10:31.222517 28536 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.222659 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 149 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 148. Current term is 149. Ops: []
I20250113 21:10:31.222769 28536 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 149 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:31.222829 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 149 candidate_status { last_received { term: 148 index: 123 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:31.222885 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 148 FOLLOWER]: Advancing to term 149
I20250113 21:10:31.222919 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 149 candidate_status { last_received { term: 148 index: 123 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:31.222887 28515 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 148.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250113 21:10:31.223083 28515 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 148 LEADER]: Stepping down as leader of term 148
I20250113 21:10:31.223111 28515 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 148 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:31.223168 28515 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 148.123, Last appended by leader: 123, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.223246 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 148 FOLLOWER]: Advancing to term 149
I20250113 21:10:31.223460 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 149.
I20250113 21:10:31.223644 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 149 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:31.223730 28536 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 149 FOLLOWER]: Leader election won for term 149
I20250113 21:10:31.223775 28536 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 149 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:31.223819 28536 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 148.123, Last appended by leader: 123, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.224211 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 148 to 149, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 149 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:31.333648 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 149 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 148 index: 123. Preceding OpId from leader: term: 149 index: 124. (index mismatch)
I20250113 21:10:31.333920 28536 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250113 21:10:31.334818 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 149 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 148 index: 123. Preceding OpId from leader: term: 149 index: 124. (index mismatch)
I20250113 21:10:31.335089 28540 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250113 21:10:31.435571 28538 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 149 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:31.435667 28538 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 149 FOLLOWER]: Advancing to term 150
I20250113 21:10:31.436319 28538 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.436480 28538 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 150 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:31.436679 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:31.436689 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:31.436897 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 150 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:31.436975 28538 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 150 FOLLOWER]: Leader election lost for term 150. Reason: could not achieve majority
I20250113 21:10:31.445581 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 150 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 149. Current term is 150. Ops: []
I20250113 21:10:31.445901 28533 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 149.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250113 21:10:31.446050 28536 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 149 LEADER]: Stepping down as leader of term 149
I20250113 21:10:31.446094 28536 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 149 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:31.446157 28536 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.446249 28536 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 149 FOLLOWER]: Advancing to term 150
I20250113 21:10:31.528012 28515 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 149 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:31.528115 28515 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 149 FOLLOWER]: Advancing to term 150
I20250113 21:10:31.528774 28515 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.528920 28515 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 150 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:31.529150 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:31.529157 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:31.529307 28332 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 150 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 150: Already voted for candidate 59ff47de1d4340e8b03823c58cb98738 in this term.
I20250113 21:10:31.529819 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 150.
I20250113 21:10:31.530002 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 150 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:31.530102 28515 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 150 FOLLOWER]: Leader election won for term 150
I20250113 21:10:31.530158 28515 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 150 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:31.530215 28515 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.530625 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 149 to 150, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 150 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:31.541431 28538 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 150 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:31.541491 28538 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 150 FOLLOWER]: Advancing to term 151
I20250113 21:10:31.542058 28538 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.542173 28538 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 151 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:31.542338 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 151 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:31.542341 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 151 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:31.542419 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 150 FOLLOWER]: Advancing to term 151
I20250113 21:10:31.542930 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 151.
I20250113 21:10:31.543087 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 151 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:31.543181 28538 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 151 FOLLOWER]: Leader election won for term 151
I20250113 21:10:31.543298 28538 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 151 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:31.543344 28538 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.543802 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 150 to 151, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 151 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:31.626214 28179 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 150 LEADER]: Stepping down as leader of term 150
I20250113 21:10:31.626295 28179 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 150 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:31.626363 28179 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 150.125, Last appended by leader: 125, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.626461 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 150 FOLLOWER]: Advancing to term 151
I20250113 21:10:31.627074 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 151 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 150 index: 125. Preceding OpId from leader: term: 151 index: 125. (term mismatch)
I20250113 21:10:31.627115 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 124
I20250113 21:10:31.627133 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 150.125
I20250113 21:10:31.627156 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 151 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:31.627377 28538 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250113 21:10:31.628441 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 151 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 149 index: 124. Preceding OpId from leader: term: 151 index: 125. (index mismatch)
I20250113 21:10:31.628677 28538 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250113 21:10:31.729935 28536 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 151 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:31.730036 28536 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 151 FOLLOWER]: Advancing to term 152
I20250113 21:10:31.730692 28536 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.730865 28536 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 152 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:31.731082 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:31.731104 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:31.731278 28179 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 151 FOLLOWER]: Advancing to term 152
I20250113 21:10:31.731834 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 152.
I20250113 21:10:31.732019 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 152 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:31.732117 28536 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 152 FOLLOWER]: Leader election won for term 152
I20250113 21:10:31.732177 28536 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 152 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:31.732241 28536 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 151.125, Last appended by leader: 125, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.732781 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 151 to 152, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 152 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:31.735046 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:31.740638 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 152 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 151. Current term is 152. Ops: []
I20250113 21:10:31.740876 28542 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 151.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250113 21:10:31.740978 28542 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 76a522394a8041a18ebb4e478f89c29f log is divergent from this leader: its last log entry 152.126 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 125
I20250113 21:10:31.741029 28541 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 151 LEADER]: Stepping down as leader of term 151
I20250113 21:10:31.741084 28541 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 151 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:31.741153 28541 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 151.125, Last appended by leader: 125, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.741247 28541 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 151 FOLLOWER]: Advancing to term 152
I20250113 21:10:31.808071 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 152 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 151 index: 125. Preceding OpId from leader: term: 152 index: 126. (index mismatch)
I20250113 21:10:31.808377 28536 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250113 21:10:31.809466 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 152 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 151 index: 125. Preceding OpId from leader: term: 152 index: 126. (index mismatch)
I20250113 21:10:31.809643 28540 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250113 21:10:31.910614 28539 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 152 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:31.910703 28539 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 152 FOLLOWER]: Advancing to term 153
I20250113 21:10:31.911432 28539 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.911586 28539 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 153 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:31.911747 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:31.911760 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:31.911955 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 153 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:31.912050 28539 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 153 FOLLOWER]: Leader election lost for term 153. Reason: could not achieve majority
I20250113 21:10:31.922927 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 153 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 152. Current term is 153. Ops: []
I20250113 21:10:31.923134 28540 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 152.126, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250113 21:10:31.923218 28540 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 152 LEADER]: Stepping down as leader of term 152
I20250113 21:10:31.923245 28540 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 152 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:31.923292 28540 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 152.126, Last appended by leader: 126, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:31.923389 28540 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 152 FOLLOWER]: Advancing to term 153
I20250113 21:10:32.003903 28541 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 152 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:32.003994 28541 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 152 FOLLOWER]: Advancing to term 153
I20250113 21:10:32.004666 28541 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.004868 28541 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 153 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:32.005079 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:32.005079 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:32.005211 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 153 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 153: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:32.005726 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 153.
I20250113 21:10:32.005923 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 153 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:32.006006 28541 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 153 FOLLOWER]: Leader election won for term 153
I20250113 21:10:32.006071 28541 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 153 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:32.006141 28541 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 152.126, Last appended by leader: 126, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.006596 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 152 to 153, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 153 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:32.060433 28539 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 153 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:32.060539 28539 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 153 FOLLOWER]: Advancing to term 154
I20250113 21:10:32.061235 28539 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.061400 28539 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 154 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:32.061614 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 154 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:32.061622 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 154 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:32.061724 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 153 FOLLOWER]: Advancing to term 154
I20250113 21:10:32.062289 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 154.
I20250113 21:10:32.062481 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 154 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:32.062567 28539 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 154 FOLLOWER]: Leader election won for term 154
I20250113 21:10:32.062614 28539 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 154 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:32.062673 28539 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 152.126, Last appended by leader: 126, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.063194 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 153 to 154, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 154 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:32.090018 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 154 LEADER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 153. Current term is 154. Ops: []
I20250113 21:10:32.090288 28541 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250113 21:10:32.090399 28541 consensus_queue.cc:1225] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer 571116d897274e4f98bcd671bf0bfe3b log is divergent from this leader: its last log entry 154.127 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 126
I20250113 21:10:32.090446 28542 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 153 LEADER]: Stepping down as leader of term 153
I20250113 21:10:32.090512 28542 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 153 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:32.090579 28542 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 153.127, Last appended by leader: 127, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.090670 28542 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 153 FOLLOWER]: Advancing to term 154
I20250113 21:10:32.145960 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 154 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 153 index: 127. Preceding OpId from leader: term: 154 index: 127. (term mismatch)
I20250113 21:10:32.146034 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 126
I20250113 21:10:32.146046 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 153.127
I20250113 21:10:32.146085 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 154 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:32.146310 28539 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250113 21:10:32.147290 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 154 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 152 index: 126. Preceding OpId from leader: term: 154 index: 127. (index mismatch)
I20250113 21:10:32.147517 28539 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250113 21:10:32.500099 28540 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 154 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:32.500205 28540 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 154 FOLLOWER]: Advancing to term 155
I20250113 21:10:32.500922 28540 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.501117 28540 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 155 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:32.501329 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:32.501329 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:32.501585 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 155 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:32.501691 28540 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 155 FOLLOWER]: Leader election lost for term 155. Reason: could not achieve majority
I20250113 21:10:32.504537 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 155 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 154. Current term is 155. Ops: []
I20250113 21:10:32.504739 28543 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 154.127, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250113 21:10:32.504858 28543 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 154 LEADER]: Stepping down as leader of term 154
I20250113 21:10:32.504891 28543 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 154 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:32.504947 28543 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 154.127, Last appended by leader: 127, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.505028 28543 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 154 FOLLOWER]: Advancing to term 155
I20250113 21:10:32.534932 28542 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 154 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:32.535022 28542 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 154 FOLLOWER]: Advancing to term 155
I20250113 21:10:32.535637 28542 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.535782 28542 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 155 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:32.535976 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:32.535981 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:32.536098 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 155: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:32.536586 28179 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 155.
I20250113 21:10:32.536756 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 155 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:32.536849 28542 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 155 FOLLOWER]: Leader election won for term 155
I20250113 21:10:32.536906 28542 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 155 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:32.536983 28542 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 154.127, Last appended by leader: 127, Current term: 155, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.537401 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 154 to 155, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 155 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:32.629945 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 155 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 154 index: 127. Preceding OpId from leader: term: 155 index: 128. (index mismatch)
I20250113 21:10:32.630216 28542 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250113 21:10:32.631295 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 155 FOLLOWER]: Refusing update from remote peer 59ff47de1d4340e8b03823c58cb98738: Log matching property violated. Preceding OpId in replica: term: 154 index: 127. Preceding OpId from leader: term: 155 index: 128. (index mismatch)
I20250113 21:10:32.631538 28541 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250113 21:10:32.735255 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:32.832065 28540 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 155 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:32.832165 28540 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 155 FOLLOWER]: Advancing to term 156
I20250113 21:10:32.832832 28540 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.833011 28540 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 156 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:32.833215 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:32.833215 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:32.833493 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 156 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:32.833585 28540 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 156 FOLLOWER]: Leader election lost for term 156. Reason: could not achieve majority
I20250113 21:10:32.854559 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 156 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 155. Current term is 156. Ops: []
I20250113 21:10:32.854833 28542 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 155.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250113 21:10:32.854954 28542 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 155 LEADER]: Stepping down as leader of term 155
I20250113 21:10:32.854988 28542 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 155 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:32.855036 28542 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 155.128, Last appended by leader: 128, Current term: 155, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.855111 28542 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 155 FOLLOWER]: Advancing to term 156
I20250113 21:10:32.918084 28543 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 155 FOLLOWER]: Starting leader election (detected failure of leader 59ff47de1d4340e8b03823c58cb98738)
I20250113 21:10:32.918175 28543 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 155 FOLLOWER]: Advancing to term 156
I20250113 21:10:32.918869 28543 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.919067 28543 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 156 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:32.919270 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:32.919279 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:32.919412 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 156 FOLLOWER]: Leader election vote request: Denying vote to candidate 571116d897274e4f98bcd671bf0bfe3b in current term 156: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
I20250113 21:10:32.919939 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 156.
I20250113 21:10:32.920106 28145 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 156 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738; no voters: 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:32.920192 28543 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 156 FOLLOWER]: Leader election won for term 156
I20250113 21:10:32.920244 28543 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 156 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:32.920296 28543 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 155.128, Last appended by leader: 128, Current term: 156, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.920722 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 155 to 156, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 156 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:32.972168 28540 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 156 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:32.972254 28540 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 156 FOLLOWER]: Advancing to term 157
I20250113 21:10:32.972877 28540 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.973014 28540 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 157 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:32.973227 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 157 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:32.973232 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 157 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:32.973347 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 156 FOLLOWER]: Advancing to term 157
I20250113 21:10:32.973896 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 157.
I20250113 21:10:32.974050 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:32.974128 28540 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 157 FOLLOWER]: Leader election won for term 157
I20250113 21:10:32.974257 28540 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 157 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:32.974314 28540 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 155.128, Last appended by leader: 128, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:32.974743 28075 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 156 to 157, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 157 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:32.999965 28332 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 157 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 156. Current term is 157. Ops: []
I20250113 21:10:33.000300 28543 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: INVALID_TERM, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250113 21:10:33.000414 28543 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 156 LEADER]: Stepping down as leader of term 156
I20250113 21:10:33.000452 28543 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 156 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:33.000504 28543 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 156.129, Last appended by leader: 129, Current term: 156, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.000576 28543 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 156 FOLLOWER]: Advancing to term 157
I20250113 21:10:33.066157 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 157 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 155 index: 128. Preceding OpId from leader: term: 157 index: 129. (index mismatch)
I20250113 21:10:33.066437 28540 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250113 21:10:33.067449 28179 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 157 FOLLOWER]: Refusing update from remote peer 76a522394a8041a18ebb4e478f89c29f: Log matching property violated. Preceding OpId in replica: term: 156 index: 129. Preceding OpId from leader: term: 157 index: 129. (term mismatch)
I20250113 21:10:33.067503 28179 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting all ops after (but not including) 128
I20250113 21:10:33.067524 28179 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: Aborting uncommitted NO_OP operation due to leader change: 156.129
I20250113 21:10:33.067552 28179 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 157 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:33.067724 28540 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250113 21:10:33.277899 28543 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 157 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:33.278002 28543 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 157 FOLLOWER]: Advancing to term 158
I20250113 21:10:33.278666 28543 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.278825 28543 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 158 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:33.279016 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:33.279026 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:33.279292 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 158 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b; no voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:33.279381 28543 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 158 FOLLOWER]: Leader election lost for term 158. Reason: could not achieve majority
I20250113 21:10:33.290427 28179 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 158 FOLLOWER]: Rejecting Update request from peer 76a522394a8041a18ebb4e478f89c29f for earlier term 157. Current term is 158. Ops: []
I20250113 21:10:33.290683 28546 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 }, Status: INVALID_TERM, Last received: 157.129, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250113 21:10:33.290791 28546 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 157 LEADER]: Stepping down as leader of term 157
I20250113 21:10:33.290822 28546 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 157 LEADER]: Becoming Follower/Learner. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:33.290877 28546 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 157.129, Last appended by leader: 129, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.290958 28546 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 157 FOLLOWER]: Advancing to term 158
I20250113 21:10:33.362468 28542 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 157 FOLLOWER]: Starting leader election (detected failure of leader 76a522394a8041a18ebb4e478f89c29f)
I20250113 21:10:33.362555 28542 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 157 FOLLOWER]: Advancing to term 158
I20250113 21:10:33.363164 28542 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.363340 28542 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 158 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:33.363583 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:33.363583 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:33.363693 28179 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 158 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 158: Already voted for candidate 571116d897274e4f98bcd671bf0bfe3b in this term.
I20250113 21:10:33.364288 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ff47de1d4340e8b03823c58cb98738 in term 158.
I20250113 21:10:33.364455 28295 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 158 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:33.364532 28542 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 158 FOLLOWER]: Leader election won for term 158
I20250113 21:10:33.364627 28542 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 158 LEADER]: Becoming Leader. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:33.364667 28542 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 157.129, Last appended by leader: 129, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.365104 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 reported cstate change: term changed from 157 to 158, leader changed from 76a522394a8041a18ebb4e478f89c29f (127.27.99.66) to 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67). New cstate: current_term: 158 leader_uuid: "59ff47de1d4340e8b03823c58cb98738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:33.396054 28543 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 158 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:33.396142 28543 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 158 FOLLOWER]: Advancing to term 159
I20250113 21:10:33.396755 28543 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.396914 28543 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 159 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759)
I20250113 21:10:33.397159 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 159 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:33.397230 28256 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 158 FOLLOWER]: Advancing to term 159
I20250113 21:10:33.397174 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "571116d897274e4f98bcd671bf0bfe3b" candidate_term: 159 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:33.397768 28256 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 159 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571116d897274e4f98bcd671bf0bfe3b in term 159.
I20250113 21:10:33.397939 28143 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [CANDIDATE]: Term 159 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f; no voters: 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:33.398026 28543 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 159 FOLLOWER]: Leader election won for term 159
I20250113 21:10:33.398078 28543 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 159 LEADER]: Becoming Leader. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:33.398123 28543 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 157.129, Last appended by leader: 129, Current term: 159, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.398576 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b reported cstate change: term changed from 158 to 159, leader changed from 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67) to 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65). New cstate: current_term: 159 leader_uuid: "571116d897274e4f98bcd671bf0bfe3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:33.448475 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 159 FOLLOWER]: Rejecting Update request from peer 59ff47de1d4340e8b03823c58cb98738 for earlier term 158. Current term is 159. Ops: []
I20250113 21:10:33.448762 28542 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250113 21:10:33.448874 28542 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 158 LEADER]: Stepping down as leader of term 158
I20250113 21:10:33.448912 28542 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 158 LEADER]: Becoming Follower/Learner. State: Replica: 59ff47de1d4340e8b03823c58cb98738, State: Running, Role: LEADER
I20250113 21:10:33.448969 28542 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 158.130, Last appended by leader: 130, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.449029 28542 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 158 FOLLOWER]: Advancing to term 159
I20250113 21:10:33.482986 28332 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 159 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 158 index: 130. Preceding OpId from leader: term: 159 index: 130. (term mismatch)
I20250113 21:10:33.483067 28332 pending_rounds.cc:77] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting all ops after (but not including) 129
I20250113 21:10:33.483080 28332 pending_rounds.cc:99] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: Aborting uncommitted NO_OP operation due to leader change: 158.130
I20250113 21:10:33.483095 28332 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 159 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:33.483321 28543 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250113 21:10:33.484277 28256 raft_consensus.cc:1270] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 159 FOLLOWER]: Refusing update from remote peer 571116d897274e4f98bcd671bf0bfe3b: Log matching property violated. Preceding OpId in replica: term: 157 index: 129. Preceding OpId from leader: term: 159 index: 130. (index mismatch)
I20250113 21:10:33.484476 28515 consensus_queue.cc:1035] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250113 21:10:33.665810 28545 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 159 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:33.665937 28545 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 159 FOLLOWER]: Advancing to term 160
I20250113 21:10:33.666679 28545 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.666846 28545 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 160 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:33.667002 28179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 160 candidate_status { last_received { term: 159 index: 130 } } ignore_live_leader: false dest_uuid: "571116d897274e4f98bcd671bf0bfe3b"
I20250113 21:10:33.666982 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 160 candidate_status { last_received { term: 159 index: 130 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:33.667230 28223 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 160 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b, 59ff47de1d4340e8b03823c58cb98738
I20250113 21:10:33.667311 28545 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 160 FOLLOWER]: Leader election lost for term 160. Reason: could not achieve majority
I20250113 21:10:33.672461 28256 raft_consensus.cc:1235] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 160 FOLLOWER]: Rejecting Update request from peer 571116d897274e4f98bcd671bf0bfe3b for earlier term 159. Current term is 160. Ops: []
I20250113 21:10:33.672696 28548 consensus_queue.cc:1046] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 }, Status: INVALID_TERM, Last received: 159.130, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20250113 21:10:33.672812 28548 raft_consensus.cc:3049] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 159 LEADER]: Stepping down as leader of term 159
I20250113 21:10:33.672847 28548 raft_consensus.cc:738] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 159 LEADER]: Becoming Follower/Learner. State: Replica: 571116d897274e4f98bcd671bf0bfe3b, State: Running, Role: LEADER
I20250113 21:10:33.672909 28548 consensus_queue.cc:260] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 159.130, Last appended by leader: 130, Current term: 159, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.672991 28548 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 159 FOLLOWER]: Advancing to term 160
I20250113 21:10:33.735486 28045 timestamp_advancement-itest.cc:261] GCing logs...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:261: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:326: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250113 21:10:33.735730 28045 tablet_server.cc:178] TabletServer@127.27.99.65:0 shutting down...
I20250113 21:10:33.764756 28542 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 159 FOLLOWER]: Starting leader election (detected failure of leader 571116d897274e4f98bcd671bf0bfe3b)
I20250113 21:10:33.764839 28542 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 159 FOLLOWER]: Advancing to term 160
I20250113 21:10:33.765479 28542 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.765646 28542 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 160 election: Requested vote from peers 76a522394a8041a18ebb4e478f89c29f (127.27.99.66:38759), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:33.765842 28256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "59ff47de1d4340e8b03823c58cb98738" candidate_term: 160 candidate_status { last_received { term: 159 index: 130 } } ignore_live_leader: false dest_uuid: "76a522394a8041a18ebb4e478f89c29f"
I20250113 21:10:33.765962 28256 raft_consensus.cc:2388] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 160 FOLLOWER]: Leader election vote request: Denying vote to candidate 59ff47de1d4340e8b03823c58cb98738 in current term 160: Already voted for candidate 76a522394a8041a18ebb4e478f89c29f in this term.
W20250113 21:10:33.765998 28298 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250113 21:10:33.766463 28298 leader_election.cc:336] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 160 election: RPC error from VoteRequest() call to peer 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250113 21:10:33.766510 28298 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [CANDIDATE]: Term 160 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738; no voters: 571116d897274e4f98bcd671bf0bfe3b, 76a522394a8041a18ebb4e478f89c29f
I20250113 21:10:33.766592 28542 raft_consensus.cc:2743] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 160 FOLLOWER]: Leader election lost for term 160. Reason: could not achieve majority
I20250113 21:10:33.807732 28545 raft_consensus.cc:491] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 160 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:33.807838 28545 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 160 FOLLOWER]: Advancing to term 161
I20250113 21:10:33.808593 28545 raft_consensus.cc:513] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.808768 28545 leader_election.cc:290] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 161 election: Requested vote from peers 59ff47de1d4340e8b03823c58cb98738 (127.27.99.67:45391), 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365)
I20250113 21:10:33.808966 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ecfe918cd2f49cfa3f0e1bf8d643e6a" candidate_uuid: "76a522394a8041a18ebb4e478f89c29f" candidate_term: 161 candidate_status { last_received { term: 159 index: 130 } } ignore_live_leader: false dest_uuid: "59ff47de1d4340e8b03823c58cb98738"
I20250113 21:10:33.809038 28332 raft_consensus.cc:3054] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 160 FOLLOWER]: Advancing to term 161
W20250113 21:10:33.809263 28223 leader_election.cc:336] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 161 election: RPC error from VoteRequest() call to peer 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65:43365): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250113 21:10:33.809612 28332 raft_consensus.cc:2463] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a522394a8041a18ebb4e478f89c29f in term 161.
I20250113 21:10:33.809901 28222 leader_election.cc:304] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [CANDIDATE]: Term 161 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59ff47de1d4340e8b03823c58cb98738, 76a522394a8041a18ebb4e478f89c29f; no voters: 571116d897274e4f98bcd671bf0bfe3b
I20250113 21:10:33.809994 28545 raft_consensus.cc:2798] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 161 FOLLOWER]: Leader election won for term 161
I20250113 21:10:33.810062 28545 raft_consensus.cc:695] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 161 LEADER]: Becoming Leader. State: Replica: 76a522394a8041a18ebb4e478f89c29f, State: Running, Role: LEADER
I20250113 21:10:33.810118 28545 consensus_queue.cc:237] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 159.130, Last appended by leader: 130, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } }
I20250113 21:10:33.810600 28073 catalog_manager.cc:5526] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f reported cstate change: term changed from 159 to 161, leader changed from 571116d897274e4f98bcd671bf0bfe3b (127.27.99.65) to 76a522394a8041a18ebb4e478f89c29f (127.27.99.66). New cstate: current_term: 161 leader_uuid: "76a522394a8041a18ebb4e478f89c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ff47de1d4340e8b03823c58cb98738" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a522394a8041a18ebb4e478f89c29f" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571116d897274e4f98bcd671bf0bfe3b" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 43365 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:33.890780 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:33.890971 28045 tablet_replica.cc:331] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b: stopping tablet replica
I20250113 21:10:33.891050 28045 raft_consensus.cc:2238] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 160 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:33.891124 28045 raft_consensus.cc:2267] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 571116d897274e4f98bcd671bf0bfe3b [term 160 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:33.904105 28045 tablet_server.cc:195] TabletServer@127.27.99.65:0 shutdown complete.
I20250113 21:10:33.905246 28045 tablet_server.cc:178] TabletServer@127.27.99.66:0 shutting down...
I20250113 21:10:33.916301 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:33.916432 28045 tablet_replica.cc:331] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: stopping tablet replica
I20250113 21:10:33.916497 28045 raft_consensus.cc:2238] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 161 LEADER]: Raft consensus shutting down.
I20250113 21:10:33.916565 28045 pending_rounds.cc:62] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Trying to abort 1 pending ops.
I20250113 21:10:33.916595 28045 pending_rounds.cc:69] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f: Aborting op as it isn't in flight: id { term: 161 index: 131 } timestamp: 7113943588086820864 op_type: NO_OP noop_request { }
I20250113 21:10:33.916664 28045 raft_consensus.cc:2883] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 161 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 21:10:33.916695 28045 raft_consensus.cc:2267] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 76a522394a8041a18ebb4e478f89c29f [term 161 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:33.928562 28045 tablet_server.cc:195] TabletServer@127.27.99.66:0 shutdown complete.
I20250113 21:10:33.929363 28045 tablet_server.cc:178] TabletServer@127.27.99.67:0 shutting down...
I20250113 21:10:33.936170 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:33.936280 28045 tablet_replica.cc:331] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738: stopping tablet replica
I20250113 21:10:33.936352 28045 raft_consensus.cc:2238] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 161 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:33.936405 28045 raft_consensus.cc:2267] T 3ecfe918cd2f49cfa3f0e1bf8d643e6a P 59ff47de1d4340e8b03823c58cb98738 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:33.948668 28045 tablet_server.cc:195] TabletServer@127.27.99.67:0 shutdown complete.
I20250113 21:10:33.949535 28045 master.cc:537] Master@127.27.99.126:45585 shutting down...
I20250113 21:10:34.067391 28045 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 1 LEADER]: Raft consensus shutting down.
I20250113 21:10:34.067505 28045 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:34.067533 28045 tablet_replica.cc:331] T 00000000000000000000000000000000 P ef42ff0cfe3149a8a27e0757e790232d: stopping tablet replica
I20250113 21:10:34.079468 28045 master.cc:559] Master@127.27.99.126:45585 shutdown complete.
I20250113 21:10:34.081262 28045 test_util.cc:181] -----------------------------------------------
I20250113 21:10:34.081303 28045 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736802601520715-28045-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (32557 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20250113 21:10:34.081750 28045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.99.126:43963
I20250113 21:10:34.081926 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:34.084101 28552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:34.084702 28551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:34.085363 28554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:34.086023 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:34.086088 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:34.086114 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:34.086126 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802634086127 us; error 0 us; skew 500 ppm
I20250113 21:10:34.091147 28045 webserver.cc:458] Webserver started at http://127.27.99.126:32857/ using document root <none> and password file <none>
I20250113 21:10:34.091218 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:34.091253 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:34.091280 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:34.091508 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/master-0-root/instance:
uuid: "cacd249002a647df99098ed657565b78"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.092298 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:34.092723 28559 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.092841 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.092890 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/master-0-root
uuid: "cacd249002a647df99098ed657565b78"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.092924 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:34.106266 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:34.106412 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:34.108778 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.126:43963
I20250113 21:10:34.110563 28611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.126:43963 every 8 connection(s)
I20250113 21:10:34.110662 28612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:34.111591 28612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78: Bootstrap starting.
I20250113 21:10:34.111886 28612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:34.112324 28612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78: No bootstrap required, opened a new log
I20250113 21:10:34.112445 28612 raft_consensus.cc:357] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacd249002a647df99098ed657565b78" member_type: VOTER }
I20250113 21:10:34.112494 28612 raft_consensus.cc:383] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:34.112519 28612 raft_consensus.cc:738] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacd249002a647df99098ed657565b78, State: Initialized, Role: FOLLOWER
I20250113 21:10:34.112566 28612 consensus_queue.cc:260] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacd249002a647df99098ed657565b78" member_type: VOTER }
I20250113 21:10:34.112608 28612 raft_consensus.cc:397] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 21:10:34.112638 28612 raft_consensus.cc:491] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 21:10:34.112668 28612 raft_consensus.cc:3054] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:34.113117 28612 raft_consensus.cc:513] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacd249002a647df99098ed657565b78" member_type: VOTER }
I20250113 21:10:34.113170 28612 leader_election.cc:304] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cacd249002a647df99098ed657565b78; no voters:
I20250113 21:10:34.113277 28612 leader_election.cc:290] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 21:10:34.113309 28615 raft_consensus.cc:2798] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:34.113416 28612 sys_catalog.cc:564] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 21:10:34.113440 28615 raft_consensus.cc:695] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 1 LEADER]: Becoming Leader. State: Replica: cacd249002a647df99098ed657565b78, State: Running, Role: LEADER
I20250113 21:10:34.113487 28615 consensus_queue.cc:237] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacd249002a647df99098ed657565b78" member_type: VOTER }
I20250113 21:10:34.113708 28616 sys_catalog.cc:455] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cacd249002a647df99098ed657565b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacd249002a647df99098ed657565b78" member_type: VOTER } }
I20250113 21:10:34.113723 28617 sys_catalog.cc:455] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cacd249002a647df99098ed657565b78. Latest consensus state: current_term: 1 leader_uuid: "cacd249002a647df99098ed657565b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacd249002a647df99098ed657565b78" member_type: VOTER } }
I20250113 21:10:34.113795 28617 sys_catalog.cc:458] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:34.113961 28616 sys_catalog.cc:458] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:34.114109 28621 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 21:10:34.114266 28621 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 21:10:34.114665 28045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 21:10:34.114889 28621 catalog_manager.cc:1348] Generated new cluster ID: 4b0ced7a223f42608ada8ffc157c4630
I20250113 21:10:34.114935 28621 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 21:10:34.129348 28621 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 21:10:34.129750 28621 catalog_manager.cc:1505] Loading token signing keys...
I20250113 21:10:34.143883 28621 catalog_manager.cc:5899] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78: Generated new TSK 0
I20250113 21:10:34.143970 28621 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 21:10:34.146450 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:34.148545 28633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:34.149293 28634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:34.149982 28636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:34.150696 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:34.150812 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:34.150840 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:34.150853 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802634150853 us; error 0 us; skew 500 ppm
I20250113 21:10:34.151494 28045 webserver.cc:458] Webserver started at http://127.27.99.65:33161/ using document root <none> and password file <none>
I20250113 21:10:34.151562 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:34.151597 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:34.151629 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:34.151882 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root/instance:
uuid: "ea17415e1051464c83d233ab0fa803c3"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.152676 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:34.153091 28641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.153213 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.153252 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root
uuid: "ea17415e1051464c83d233ab0fa803c3"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.153298 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:34.158016 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:34.158164 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:34.158290 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:34.158469 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:34.158496 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.158533 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:34.158557 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.161288 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.65:40911
I20250113 21:10:34.161322 28705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.65:40911 every 8 connection(s)
I20250113 21:10:34.161602 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:34.163343 28709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:34.163960 28713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:34.164691 28045 server_base.cc:1034] running on GCE node
W20250113 21:10:34.165129 28710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:34.165158 28706 heartbeater.cc:346] Connected to a master server at 127.27.99.126:43963
I20250113 21:10:34.165256 28706 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:34.165346 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 21:10:34.165367 28706 heartbeater.cc:510] Master 127.27.99.126:43963 requested a full tablet report, sending...
W20250113 21:10:34.165387 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:34.165400 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802634165401 us; error 0 us; skew 500 ppm
I20250113 21:10:34.165629 28575 ts_manager.cc:194] Registered new tserver with Master: ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911)
I20250113 21:10:34.166222 28575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47238
I20250113 21:10:34.166515 28045 webserver.cc:458] Webserver started at http://127.27.99.66:45699/ using document root <none> and password file <none>
I20250113 21:10:34.166599 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:34.166644 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:34.166689 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:34.166927 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-1-root/instance:
uuid: "85ea282909514f9090fe51fe85389b6a"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.167728 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:34.168118 28718 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.168252 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.168291 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-1-root
uuid: "85ea282909514f9090fe51fe85389b6a"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.168336 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:34.182189 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:34.182343 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:34.182466 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:34.182629 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:34.182654 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.182682 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:34.182706 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.185396 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.66:40939
I20250113 21:10:34.185456 28782 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.66:40939 every 8 connection(s)
I20250113 21:10:34.185736 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:34.187664 28786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:34.188256 28787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:34.188486 28783 heartbeater.cc:346] Connected to a master server at 127.27.99.126:43963
I20250113 21:10:34.188553 28783 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:34.188623 28783 heartbeater.cc:510] Master 127.27.99.126:43963 requested a full tablet report, sending...
I20250113 21:10:34.188812 28575 ts_manager.cc:194] Registered new tserver with Master: 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
W20250113 21:10:34.188858 28789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:34.189127 28575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47250
I20250113 21:10:34.189500 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:34.189607 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:34.189633 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:34.189646 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802634189645 us; error 0 us; skew 500 ppm
I20250113 21:10:34.190258 28045 webserver.cc:458] Webserver started at http://127.27.99.67:46083/ using document root <none> and password file <none>
I20250113 21:10:34.190321 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:34.190351 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:34.190387 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:34.190618 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-2-root/instance:
uuid: "162d4645e5644b3aa05e447cf9ddf677"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.191499 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:34.191883 28794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.192003 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.192046 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-2-root
uuid: "162d4645e5644b3aa05e447cf9ddf677"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:34.192090 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:34.196848 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:34.197014 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:34.197141 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:34.197321 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:34.197350 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.197376 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:34.197402 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.200610 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.67:38113
I20250113 21:10:34.200644 28857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.67:38113 every 8 connection(s)
I20250113 21:10:34.203287 28858 heartbeater.cc:346] Connected to a master server at 127.27.99.126:43963
I20250113 21:10:34.203342 28858 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:34.203526 28858 heartbeater.cc:510] Master 127.27.99.126:43963 requested a full tablet report, sending...
I20250113 21:10:34.203779 28575 ts_manager.cc:194] Registered new tserver with Master: 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113)
I20250113 21:10:34.204070 28575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47256
I20250113 21:10:34.204120 28045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003339335s
I20250113 21:10:34.210907 28045 test_util.cc:274] Using random seed: -535912870
I20250113 21:10:34.215324 28575 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47284:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 21:10:34.215447 28575 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 21:10:34.220008 28822 tablet_service.cc:1467] Processing CreateTablet for tablet b4d8bcc498774b26a957a97b8986f203 (DEFAULT_TABLE table=test-workload [id=19ae2ae07057475398da1a907dfb9d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:34.220145 28822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d8bcc498774b26a957a97b8986f203. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:34.220207 28669 tablet_service.cc:1467] Processing CreateTablet for tablet b4d8bcc498774b26a957a97b8986f203 (DEFAULT_TABLE table=test-workload [id=19ae2ae07057475398da1a907dfb9d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:34.220289 28669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d8bcc498774b26a957a97b8986f203. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:34.220407 28746 tablet_service.cc:1467] Processing CreateTablet for tablet b4d8bcc498774b26a957a97b8986f203 (DEFAULT_TABLE table=test-workload [id=19ae2ae07057475398da1a907dfb9d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:34.220504 28746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d8bcc498774b26a957a97b8986f203. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:34.221619 28881 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: Bootstrap starting.
I20250113 21:10:34.221848 28879 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a: Bootstrap starting.
I20250113 21:10:34.221943 28880 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Bootstrap starting.
I20250113 21:10:34.222133 28881 tablet_bootstrap.cc:654] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:34.222381 28879 tablet_bootstrap.cc:654] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:34.222381 28880 tablet_bootstrap.cc:654] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:34.222676 28881 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: No bootstrap required, opened a new log
I20250113 21:10:34.222716 28881 ts_tablet_manager.cc:1397] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:34.222873 28881 raft_consensus.cc:357] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.222936 28879 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a: No bootstrap required, opened a new log
I20250113 21:10:34.222995 28879 ts_tablet_manager.cc:1397] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:34.223160 28880 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: No bootstrap required, opened a new log
I20250113 21:10:34.223126 28879 raft_consensus.cc:357] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.223187 28879 raft_consensus.cc:383] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:34.223191 28880 ts_tablet_manager.cc:1397] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:34.223222 28879 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ea282909514f9090fe51fe85389b6a, State: Initialized, Role: FOLLOWER
I20250113 21:10:34.223289 28879 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.223341 28880 raft_consensus.cc:357] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.223404 28880 raft_consensus.cc:383] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:34.223423 28880 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea17415e1051464c83d233ab0fa803c3, State: Initialized, Role: FOLLOWER
I20250113 21:10:34.223434 28879 ts_tablet_manager.cc:1428] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.223467 28880 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.223470 28783 heartbeater.cc:502] Master 127.27.99.126:43963 was elected leader, sending a full tablet report...
I20250113 21:10:34.222937 28881 raft_consensus.cc:383] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:34.223603 28881 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162d4645e5644b3aa05e447cf9ddf677, State: Initialized, Role: FOLLOWER
I20250113 21:10:34.223618 28880 ts_tablet_manager.cc:1428] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:34.223677 28706 heartbeater.cc:502] Master 127.27.99.126:43963 was elected leader, sending a full tablet report...
I20250113 21:10:34.223655 28881 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.223811 28881 ts_tablet_manager.cc:1428] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:34.223874 28858 heartbeater.cc:502] Master 127.27.99.126:43963 was elected leader, sending a full tablet report...
I20250113 21:10:34.254648 28887 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 21:10:34.254730 28887 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.254869 28887 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:34.257527 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3" is_pre_election: true
I20250113 21:10:34.257632 28679 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 162d4645e5644b3aa05e447cf9ddf677 in term 0.
I20250113 21:10:34.257730 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a" is_pre_election: true
I20250113 21:10:34.257820 28756 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 162d4645e5644b3aa05e447cf9ddf677 in term 0.
I20250113 21:10:34.257911 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677, ea17415e1051464c83d233ab0fa803c3; no voters:
I20250113 21:10:34.258003 28887 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 21:10:34.258044 28887 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:34.258072 28887 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:34.258674 28887 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.258771 28887 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 1 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:34.258930 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:34.258971 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:34.258998 28679 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:34.259013 28756 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:34.259608 28756 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 162d4645e5644b3aa05e447cf9ddf677 in term 1.
I20250113 21:10:34.259622 28679 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 162d4645e5644b3aa05e447cf9ddf677 in term 1.
I20250113 21:10:34.259750 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677, 85ea282909514f9090fe51fe85389b6a; no voters:
I20250113 21:10:34.259855 28887 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:34.259984 28887 raft_consensus.cc:695] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 1 LEADER]: Becoming Leader. State: Replica: 162d4645e5644b3aa05e447cf9ddf677, State: Running, Role: LEADER
I20250113 21:10:34.260030 28887 consensus_queue.cc:237] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.260600 28575 catalog_manager.cc:5526] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 reported cstate change: term changed from 0 to 1, leader changed from <none> to 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67). New cstate: current_term: 1 leader_uuid: "162d4645e5644b3aa05e447cf9ddf677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:34.265684 28756 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 1 FOLLOWER]: Refusing update from remote peer 162d4645e5644b3aa05e447cf9ddf677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 21:10:34.265748 28679 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 1 FOLLOWER]: Refusing update from remote peer 162d4645e5644b3aa05e447cf9ddf677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 21:10:34.265920 28887 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 21:10:34.266021 28889 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 21:10:34.267370 28896 mvcc.cc:204] Tried to move back new op lower bound from 7113943589951778816 to 7113943589929619456. Current Snapshot: MvccSnapshot[applied={T|T < 7113943589951197184}]
I20250113 21:10:34.568825 28885 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 162d4645e5644b3aa05e447cf9ddf677)
I20250113 21:10:34.568924 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:34.569089 28886 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 162d4645e5644b3aa05e447cf9ddf677)
I20250113 21:10:34.569166 28886 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:34.569847 28886 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.569844 28885 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.570094 28885 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 2 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113)
I20250113 21:10:34.570080 28886 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 2 election: Requested vote from peers 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:34.573473 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:34.573565 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:34.573661 28756 raft_consensus.cc:2388] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ea17415e1051464c83d233ab0fa803c3 in current term 2: Already voted for candidate 85ea282909514f9090fe51fe85389b6a in this term.
I20250113 21:10:34.573565 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:34.573724 28679 raft_consensus.cc:2388] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85ea282909514f9090fe51fe85389b6a in current term 2: Already voted for candidate ea17415e1051464c83d233ab0fa803c3 in this term.
I20250113 21:10:34.573887 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:34.573899 28721 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85ea282909514f9090fe51fe85389b6a; no voters: 162d4645e5644b3aa05e447cf9ddf677, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:34.574025 28885 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 21:10:34.574115 28644 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea17415e1051464c83d233ab0fa803c3; no voters: 162d4645e5644b3aa05e447cf9ddf677, 85ea282909514f9090fe51fe85389b6a
I20250113 21:10:34.574196 28886 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 21:10:34.577311 28679 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 2 FOLLOWER]: Rejecting Update request from peer 162d4645e5644b3aa05e447cf9ddf677 for earlier term 1. Current term is 2. Ops: []
I20250113 21:10:34.577526 28901 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: INVALID_TERM, Last received: 1.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250113 21:10:34.591552 28756 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 2 FOLLOWER]: Rejecting Update request from peer 162d4645e5644b3aa05e447cf9ddf677 for earlier term 1. Current term is 2. Ops: []
I20250113 21:10:34.591779 28889 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: INVALID_TERM, Last received: 1.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250113 21:10:34.609730 28901 raft_consensus.cc:3049] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 21:10:34.609840 28901 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 162d4645e5644b3aa05e447cf9ddf677, State: Running, Role: LEADER
I20250113 21:10:34.609954 28901 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.610080 28901 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:34.705312 28885 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:34.705426 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 2 FOLLOWER]: Advancing to term 3
I20250113 21:10:34.706166 28885 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.706346 28885 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 3 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113)
I20250113 21:10:34.706570 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 3 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:34.706575 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 3 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:34.706643 28832 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 2 FOLLOWER]: Advancing to term 3
I20250113 21:10:34.706691 28679 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 2 FOLLOWER]: Advancing to term 3
I20250113 21:10:34.707283 28679 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ea282909514f9090fe51fe85389b6a in term 3.
I20250113 21:10:34.707355 28832 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ea282909514f9090fe51fe85389b6a in term 3.
I20250113 21:10:34.707492 28722 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3; no voters:
I20250113 21:10:34.707594 28885 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 21:10:34.707705 28885 raft_consensus.cc:695] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 3 LEADER]: Becoming Leader. State: Replica: 85ea282909514f9090fe51fe85389b6a, State: Running, Role: LEADER
I20250113 21:10:34.707756 28885 consensus_queue.cc:237] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.708155 28575 catalog_manager.cc:5526] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a reported cstate change: term changed from 1 to 3, leader changed from 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67) to 85ea282909514f9090fe51fe85389b6a (127.27.99.66). New cstate: current_term: 3 leader_uuid: "85ea282909514f9090fe51fe85389b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:34.804729 28679 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 3 FOLLOWER]: Refusing update from remote peer 85ea282909514f9090fe51fe85389b6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20250113 21:10:34.805011 28885 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250113 21:10:34.826611 28832 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 3 FOLLOWER]: Refusing update from remote peer 85ea282909514f9090fe51fe85389b6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20250113 21:10:34.826839 28885 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250113 21:10:34.906957 28886 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85ea282909514f9090fe51fe85389b6a)
I20250113 21:10:34.907035 28886 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 3 FOLLOWER]: Advancing to term 4
I20250113 21:10:34.907649 28886 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:34.907785 28886 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 4 election: Requested vote from peers 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:34.907996 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:34.908010 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:34.908246 28644 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea17415e1051464c83d233ab0fa803c3; no voters: 162d4645e5644b3aa05e447cf9ddf677, 85ea282909514f9090fe51fe85389b6a
I20250113 21:10:34.908345 28886 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 21:10:34.918884 28679 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 4 FOLLOWER]: Rejecting Update request from peer 85ea282909514f9090fe51fe85389b6a for earlier term 3. Current term is 4. Ops: []
I20250113 21:10:34.919088 28923 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: INVALID_TERM, Last received: 3.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250113 21:10:35.003564 28885 raft_consensus.cc:3049] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 21:10:35.003650 28885 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 85ea282909514f9090fe51fe85389b6a, State: Running, Role: LEADER
I20250113 21:10:35.003719 28885 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.003821 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 3 FOLLOWER]: Advancing to term 4
I20250113 21:10:35.023015 28901 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85ea282909514f9090fe51fe85389b6a)
I20250113 21:10:35.023108 28901 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 3 FOLLOWER]: Advancing to term 4
I20250113 21:10:35.023778 28901 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.023914 28901 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 4 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:35.024148 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:35.024231 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:35.024264 28679 raft_consensus.cc:2388] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 162d4645e5644b3aa05e447cf9ddf677 in current term 4: Already voted for candidate ea17415e1051464c83d233ab0fa803c3 in this term.
I20250113 21:10:35.024914 28756 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 162d4645e5644b3aa05e447cf9ddf677 in term 4.
I20250113 21:10:35.025115 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677, 85ea282909514f9090fe51fe85389b6a; no voters: ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:35.025224 28901 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 21:10:35.025282 28901 raft_consensus.cc:695] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 4 LEADER]: Becoming Leader. State: Replica: 162d4645e5644b3aa05e447cf9ddf677, State: Running, Role: LEADER
I20250113 21:10:35.025326 28901 consensus_queue.cc:237] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.025844 28575 catalog_manager.cc:5526] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 reported cstate change: term changed from 3 to 4, leader changed from 85ea282909514f9090fe51fe85389b6a (127.27.99.66) to 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67). New cstate: current_term: 4 leader_uuid: "162d4645e5644b3aa05e447cf9ddf677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:35.057447 28886 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:35.057528 28886 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 4 FOLLOWER]: Advancing to term 5
I20250113 21:10:35.058184 28886 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.058315 28886 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 5 election: Requested vote from peers 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:35.058526 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:35.058521 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:35.058668 28756 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 4 FOLLOWER]: Advancing to term 5
I20250113 21:10:35.059175 28756 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea17415e1051464c83d233ab0fa803c3 in term 5.
I20250113 21:10:35.059370 28645 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3; no voters: 162d4645e5644b3aa05e447cf9ddf677
I20250113 21:10:35.059475 28886 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 5 FOLLOWER]: Leader election won for term 5
I20250113 21:10:35.059602 28886 raft_consensus.cc:695] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 5 LEADER]: Becoming Leader. State: Replica: ea17415e1051464c83d233ab0fa803c3, State: Running, Role: LEADER
I20250113 21:10:35.059643 28886 consensus_queue.cc:237] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.060088 28575 catalog_manager.cc:5526] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 reported cstate change: term changed from 4 to 5, leader changed from 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67) to ea17415e1051464c83d233ab0fa803c3 (127.27.99.65). New cstate: current_term: 5 leader_uuid: "ea17415e1051464c83d233ab0fa803c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:35.114154 28756 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 5 FOLLOWER]: Rejecting Update request from peer 162d4645e5644b3aa05e447cf9ddf677 for earlier term 4. Current term is 5. Ops: []
I20250113 21:10:35.114495 28901 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250113 21:10:35.121248 28679 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 5 LEADER]: Rejecting Update request from peer 162d4645e5644b3aa05e447cf9ddf677 for earlier term 4. Current term is 5. Ops: []
I20250113 21:10:35.121474 28889 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250113 21:10:35.121553 28889 consensus_queue.cc:1225] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Peer ea17415e1051464c83d233ab0fa803c3 log is divergent from this leader: its last log entry 5.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20250113 21:10:35.135412 28756 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 5 FOLLOWER]: Refusing update from remote peer ea17415e1051464c83d233ab0fa803c3: Log matching property violated. Preceding OpId in replica: term: 3 index: 16. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
I20250113 21:10:35.135735 28886 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250113 21:10:35.175556 28832 raft_consensus.cc:3049] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 21:10:35.175623 28832 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 162d4645e5644b3aa05e447cf9ddf677, State: Running, Role: LEADER
I20250113 21:10:35.175688 28832 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.17, Last appended by leader: 17, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.175781 28832 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 4 FOLLOWER]: Advancing to term 5
I20250113 21:10:35.176430 28832 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 5 FOLLOWER]: Refusing update from remote peer ea17415e1051464c83d233ab0fa803c3: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 5 index: 17. (term mismatch)
I20250113 21:10:35.176468 28832 pending_rounds.cc:77] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: Aborting all ops after (but not including) 16
I20250113 21:10:35.176489 28832 pending_rounds.cc:99] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: Aborting uncommitted NO_OP operation due to leader change: 4.17
I20250113 21:10:35.176509 28832 raft_consensus.cc:2883] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 21:10:35.176707 28886 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250113 21:10:35.331202 28885 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ea17415e1051464c83d233ab0fa803c3)
I20250113 21:10:35.331295 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 5 FOLLOWER]: Advancing to term 6
I20250113 21:10:35.331970 28885 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.332129 28885 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 6 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113)
I20250113 21:10:35.332329 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:35.332327 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:35.332631 28721 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85ea282909514f9090fe51fe85389b6a; no voters: 162d4645e5644b3aa05e447cf9ddf677, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:35.332715 28885 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250113 21:10:35.348212 28756 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 6 FOLLOWER]: Rejecting Update request from peer ea17415e1051464c83d233ab0fa803c3 for earlier term 5. Current term is 6. Ops: []
I20250113 21:10:35.348457 28925 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: INVALID_TERM, Last received: 5.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250113 21:10:35.425762 28924 raft_consensus.cc:3049] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 5 LEADER]: Stepping down as leader of term 5
I20250113 21:10:35.425851 28924 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ea17415e1051464c83d233ab0fa803c3, State: Running, Role: LEADER
I20250113 21:10:35.425935 28924 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 5.17, Last appended by leader: 17, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.426067 28924 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 5 FOLLOWER]: Advancing to term 6
I20250113 21:10:35.469197 28885 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:35.469318 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 6 FOLLOWER]: Advancing to term 7
I20250113 21:10:35.470058 28885 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.470213 28885 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 7 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113)
I20250113 21:10:35.470439 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 7 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:35.470467 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 7 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:35.470562 28679 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 6 FOLLOWER]: Advancing to term 7
I20250113 21:10:35.471094 28679 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ea282909514f9090fe51fe85389b6a in term 7.
I20250113 21:10:35.471290 28722 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3; no voters: 162d4645e5644b3aa05e447cf9ddf677
I20250113 21:10:35.471385 28885 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 7 FOLLOWER]: Leader election won for term 7
I20250113 21:10:35.471457 28885 raft_consensus.cc:695] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 7 LEADER]: Becoming Leader. State: Replica: 85ea282909514f9090fe51fe85389b6a, State: Running, Role: LEADER
I20250113 21:10:35.471520 28885 consensus_queue.cc:237] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 5.17, Last appended by leader: 17, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.472067 28576 catalog_manager.cc:5526] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a reported cstate change: term changed from 5 to 7, leader changed from ea17415e1051464c83d233ab0fa803c3 (127.27.99.65) to 85ea282909514f9090fe51fe85389b6a (127.27.99.66). New cstate: current_term: 7 leader_uuid: "85ea282909514f9090fe51fe85389b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:35.488749 28901 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ea17415e1051464c83d233ab0fa803c3)
I20250113 21:10:35.488823 28901 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 5 FOLLOWER]: Advancing to term 6
I20250113 21:10:35.489405 28901 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.489526 28901 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 6 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:35.489748 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:35.489773 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:35.489817 28679 raft_consensus.cc:2363] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 162d4645e5644b3aa05e447cf9ddf677 for earlier term 6. Current term is 7.
I20250113 21:10:35.490041 28798 leader_election.cc:400] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 6 election: Vote denied by peer ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911) with higher term. Message: Invalid argument: T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 162d4645e5644b3aa05e447cf9ddf677 for earlier term 6. Current term is 7.
I20250113 21:10:35.490075 28798 leader_election.cc:403] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250113 21:10:35.490223 28901 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 6 FOLLOWER]: Advancing to term 7
I20250113 21:10:35.490739 28901 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911) with higher term. Message: Invalid argument: T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 162d4645e5644b3aa05e447cf9ddf677 for earlier term 6. Current term is 7.
I20250113 21:10:35.549626 28832 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 7 FOLLOWER]: Refusing update from remote peer 85ea282909514f9090fe51fe85389b6a: Log matching property violated. Preceding OpId in replica: term: 5 index: 17. Preceding OpId from leader: term: 7 index: 18. (index mismatch)
I20250113 21:10:35.549961 28885 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250113 21:10:35.573186 28679 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 7 FOLLOWER]: Refusing update from remote peer 85ea282909514f9090fe51fe85389b6a: Log matching property violated. Preceding OpId in replica: term: 5 index: 17. Preceding OpId from leader: term: 7 index: 18. (index mismatch)
I20250113 21:10:35.573449 28926 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250113 21:10:35.651295 28901 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 85ea282909514f9090fe51fe85389b6a)
I20250113 21:10:35.651391 28901 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 7 FOLLOWER]: Advancing to term 8
I20250113 21:10:35.652036 28901 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.652197 28901 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 8 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:35.652418 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:35.652446 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:35.652892 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677; no voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:35.653008 28901 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 21:10:35.659943 28832 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 8 FOLLOWER]: Rejecting Update request from peer 85ea282909514f9090fe51fe85389b6a for earlier term 7. Current term is 8. Ops: []
I20250113 21:10:35.660159 28926 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 }, Status: INVALID_TERM, Last received: 7.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250113 21:10:35.757373 28901 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:35.757463 28901 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 8 FOLLOWER]: Advancing to term 9
I20250113 21:10:35.758131 28901 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.758280 28901 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 9 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:35.758471 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 9 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:35.758518 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 9 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:35.758704 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677; no voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:35.758785 28901 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250113 21:10:35.766217 28832 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Rejecting Update request from peer 85ea282909514f9090fe51fe85389b6a for earlier term 7. Current term is 9. Ops: []
I20250113 21:10:35.766459 28919 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 }, Status: INVALID_TERM, Last received: 7.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250113 21:10:35.786473 28885 raft_consensus.cc:3049] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 7 LEADER]: Stepping down as leader of term 7
I20250113 21:10:35.786532 28885 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 85ea282909514f9090fe51fe85389b6a, State: Running, Role: LEADER
I20250113 21:10:35.786605 28885 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 7.18, Last appended by leader: 18, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.786695 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 7 FOLLOWER]: Advancing to term 8
I20250113 21:10:35.836458 28924 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 85ea282909514f9090fe51fe85389b6a)
I20250113 21:10:35.836560 28924 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 7 FOLLOWER]: Advancing to term 8
I20250113 21:10:35.837318 28924 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.837467 28924 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 8 election: Requested vote from peers 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:35.837677 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:35.837677 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:35.837798 28832 raft_consensus.cc:2363] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ea17415e1051464c83d233ab0fa803c3 for earlier term 8. Current term is 9.
I20250113 21:10:35.837985 28644 leader_election.cc:400] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 8 election: Vote denied by peer 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113) with higher term. Message: Invalid argument: T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ea17415e1051464c83d233ab0fa803c3 for earlier term 8. Current term is 9.
I20250113 21:10:35.838018 28644 leader_election.cc:403] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250113 21:10:35.838069 28924 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 8 FOLLOWER]: Advancing to term 9
I20250113 21:10:35.838384 28756 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea17415e1051464c83d233ab0fa803c3 in term 8.
I20250113 21:10:35.838613 28924 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113) with higher term. Message: Invalid argument: T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ea17415e1051464c83d233ab0fa803c3 for earlier term 8. Current term is 9.
I20250113 21:10:35.862149 28901 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:35.862237 28901 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 9 FOLLOWER]: Advancing to term 10
I20250113 21:10:35.862843 28901 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.862989 28901 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 10 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:35.863200 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 10 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:35.863241 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 10 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:35.863260 28679 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 9 FOLLOWER]: Advancing to term 10
I20250113 21:10:35.863354 28756 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 8 FOLLOWER]: Advancing to term 10
I20250113 21:10:35.863837 28679 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 162d4645e5644b3aa05e447cf9ddf677 in term 10.
I20250113 21:10:35.863919 28756 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 162d4645e5644b3aa05e447cf9ddf677 in term 10.
I20250113 21:10:35.864022 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677, ea17415e1051464c83d233ab0fa803c3; no voters:
I20250113 21:10:35.864095 28901 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 10 FOLLOWER]: Leader election won for term 10
I20250113 21:10:35.864236 28901 raft_consensus.cc:695] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 10 LEADER]: Becoming Leader. State: Replica: 162d4645e5644b3aa05e447cf9ddf677, State: Running, Role: LEADER
I20250113 21:10:35.864344 28901 consensus_queue.cc:237] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 7.18, Last appended by leader: 18, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:35.864785 28576 catalog_manager.cc:5526] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 reported cstate change: term changed from 7 to 10, leader changed from 85ea282909514f9090fe51fe85389b6a (127.27.99.66) to 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67). New cstate: current_term: 10 leader_uuid: "162d4645e5644b3aa05e447cf9ddf677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:35.954217 28679 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 10 FOLLOWER]: Refusing update from remote peer 162d4645e5644b3aa05e447cf9ddf677: Log matching property violated. Preceding OpId in replica: term: 7 index: 18. Preceding OpId from leader: term: 10 index: 19. (index mismatch)
I20250113 21:10:35.954478 28901 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250113 21:10:35.960040 28756 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 10 FOLLOWER]: Refusing update from remote peer 162d4645e5644b3aa05e447cf9ddf677: Log matching property violated. Preceding OpId in replica: term: 7 index: 18. Preceding OpId from leader: term: 10 index: 19. (index mismatch)
I20250113 21:10:35.960274 28901 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250113 21:10:36.056277 28924 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 162d4645e5644b3aa05e447cf9ddf677)
I20250113 21:10:36.056392 28924 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 10 FOLLOWER]: Advancing to term 11
I20250113 21:10:36.057044 28924 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.057238 28924 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 11 election: Requested vote from peers 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:36.057428 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:36.057428 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:36.057694 28645 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea17415e1051464c83d233ab0fa803c3; no voters: 162d4645e5644b3aa05e447cf9ddf677, 85ea282909514f9090fe51fe85389b6a
I20250113 21:10:36.057817 28924 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250113 21:10:36.059753 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.060909 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.061574 28885 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 162d4645e5644b3aa05e447cf9ddf677)
I20250113 21:10:36.061620 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 10 FOLLOWER]: Advancing to term 11
I20250113 21:10:36.062227 28885 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.062362 28885 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 11 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113)
I20250113 21:10:36.062518 28679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "ea17415e1051464c83d233ab0fa803c3"
I20250113 21:10:36.062536 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:36.062580 28679 raft_consensus.cc:2388] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 85ea282909514f9090fe51fe85389b6a in current term 11: Already voted for candidate ea17415e1051464c83d233ab0fa803c3 in this term.
I20250113 21:10:36.062748 28721 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85ea282909514f9090fe51fe85389b6a; no voters: 162d4645e5644b3aa05e447cf9ddf677, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:36.062832 28885 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250113 21:10:36.063036 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.063336 28679 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 11 FOLLOWER]: Rejecting Update request from peer 162d4645e5644b3aa05e447cf9ddf677 for earlier term 10. Current term is 11. Ops: []
I20250113 21:10:36.063562 28928 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 }, Status: INVALID_TERM, Last received: 10.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250113 21:10:36.067178 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.069273 28756 raft_consensus.cc:1235] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 11 FOLLOWER]: Rejecting Update request from peer 162d4645e5644b3aa05e447cf9ddf677 for earlier term 10. Current term is 11. Ops: []
I20250113 21:10:36.069456 28901 consensus_queue.cc:1046] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: INVALID_TERM, Last received: 10.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250113 21:10:36.075323 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.091468 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.101738 28928 raft_consensus.cc:3049] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 10 LEADER]: Stepping down as leader of term 10
I20250113 21:10:36.101786 28928 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 162d4645e5644b3aa05e447cf9ddf677, State: Running, Role: LEADER
I20250113 21:10:36.101841 28928 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 10.19, Last appended by leader: 19, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.101943 28928 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 10 FOLLOWER]: Advancing to term 11
I20250113 21:10:36.123628 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.162979 28924 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:36.163064 28924 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 11 FOLLOWER]: Advancing to term 12
I20250113 21:10:36.163673 28924 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.163826 28924 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 12 election: Requested vote from peers 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:36.164021 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 12 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:36.164063 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "ea17415e1051464c83d233ab0fa803c3" candidate_term: 12 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
I20250113 21:10:36.164081 28832 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 11 FOLLOWER]: Advancing to term 12
I20250113 21:10:36.164223 28756 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 11 FOLLOWER]: Advancing to term 12
I20250113 21:10:36.164728 28832 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea17415e1051464c83d233ab0fa803c3 in term 12.
I20250113 21:10:36.164773 28756 raft_consensus.cc:2463] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea17415e1051464c83d233ab0fa803c3 in term 12.
I20250113 21:10:36.164906 28644 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677, ea17415e1051464c83d233ab0fa803c3; no voters:
I20250113 21:10:36.164999 28924 raft_consensus.cc:2798] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 FOLLOWER]: Leader election won for term 12
I20250113 21:10:36.165099 28924 raft_consensus.cc:695] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 LEADER]: Becoming Leader. State: Replica: ea17415e1051464c83d233ab0fa803c3, State: Running, Role: LEADER
I20250113 21:10:36.165158 28924 consensus_queue.cc:237] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 10.19, Last appended by leader: 19, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.165596 28576 catalog_manager.cc:5526] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 reported cstate change: term changed from 10 to 12, leader changed from 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67) to ea17415e1051464c83d233ab0fa803c3 (127.27.99.65). New cstate: current_term: 12 leader_uuid: "ea17415e1051464c83d233ab0fa803c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20250113 21:10:36.187819 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.240907 28832 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 12 FOLLOWER]: Refusing update from remote peer ea17415e1051464c83d233ab0fa803c3: Log matching property violated. Preceding OpId in replica: term: 10 index: 19. Preceding OpId from leader: term: 12 index: 20. (index mismatch)
I20250113 21:10:36.241209 28924 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250113 21:10:36.242260 28756 raft_consensus.cc:1270] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 12 FOLLOWER]: Refusing update from remote peer ea17415e1051464c83d233ab0fa803c3: Log matching property violated. Preceding OpId in replica: term: 10 index: 19. Preceding OpId from leader: term: 12 index: 20. (index mismatch)
I20250113 21:10:36.242458 28929 consensus_queue.cc:1035] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250113 21:10:36.315994 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:36.316121 28045 log_reader.cc:385] T b4d8bcc498774b26a957a97b8986f203: removed 1 log segments from log reader
I20250113 21:10:36.316147 28045 log.cc:1078] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Deleting log segment in path: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root/wals/b4d8bcc498774b26a957a97b8986f203/wal-000000001 (ops 1-15)
I20250113 21:10:36.316553 28045 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250113 21:10:36.316603 28045 master.cc:537] Master@127.27.99.126:43963 shutting down...
W20250113 21:10:36.333693 28706 heartbeater.cc:643] Failed to heartbeat to 127.27.99.126:43963 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20250113 21:10:36.358062 28045 raft_consensus.cc:2238] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 1 LEADER]: Raft consensus shutting down.
I20250113 21:10:36.358155 28045 raft_consensus.cc:2267] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:36.358175 28045 tablet_replica.cc:331] T 00000000000000000000000000000000 P cacd249002a647df99098ed657565b78: stopping tablet replica
I20250113 21:10:36.370280 28045 master.cc:559] Master@127.27.99.126:43963 shutdown complete.
I20250113 21:10:36.371532 28045 tablet_server.cc:178] TabletServer@127.27.99.65:0 shutting down...
I20250113 21:10:36.412910 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:36.413049 28045 tablet_replica.cc:331] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: stopping tablet replica
I20250113 21:10:36.413105 28045 raft_consensus.cc:2238] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 LEADER]: Raft consensus shutting down.
I20250113 21:10:36.413197 28045 raft_consensus.cc:2267] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:36.421208 28928 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ea17415e1051464c83d233ab0fa803c3)
I20250113 21:10:36.421280 28928 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 12 FOLLOWER]: Advancing to term 13
I20250113 21:10:36.421911 28928 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.422040 28928 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 13 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
I20250113 21:10:36.422385 28756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "162d4645e5644b3aa05e447cf9ddf677" candidate_term: 13 candidate_status { last_received { term: 12 index: 20 } } ignore_live_leader: false dest_uuid: "85ea282909514f9090fe51fe85389b6a"
W20250113 21:10:36.422698 28798 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911): Network error: Client connection negotiation failed: client connection to 127.27.99.65:40911: connect: Connection refused (error 111)
I20250113 21:10:36.422745 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677; no voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:36.422816 28928 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250113 21:10:36.425046 28045 tablet_server.cc:195] TabletServer@127.27.99.65:0 shutdown complete.
I20250113 21:10:36.425925 28045 tablet_server.cc:178] TabletServer@127.27.99.66:0 shutting down...
I20250113 21:10:36.434525 28885 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 12 FOLLOWER]: Starting leader election (detected failure of leader ea17415e1051464c83d233ab0fa803c3)
I20250113 21:10:36.434582 28885 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 12 FOLLOWER]: Advancing to term 13
I20250113 21:10:36.435097 28885 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.435214 28885 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 13 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113)
I20250113 21:10:36.435420 28832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d8bcc498774b26a957a97b8986f203" candidate_uuid: "85ea282909514f9090fe51fe85389b6a" candidate_term: 13 candidate_status { last_received { term: 12 index: 20 } } ignore_live_leader: false dest_uuid: "162d4645e5644b3aa05e447cf9ddf677"
I20250113 21:10:36.435482 28832 raft_consensus.cc:2388] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 85ea282909514f9090fe51fe85389b6a in current term 13: Already voted for candidate 162d4645e5644b3aa05e447cf9ddf677 in this term.
W20250113 21:10:36.435865 28722 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911): Network error: Client connection negotiation failed: client connection to 127.27.99.65:40911: connect: Connection refused (error 111)
I20250113 21:10:36.435918 28722 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85ea282909514f9090fe51fe85389b6a; no voters: 162d4645e5644b3aa05e447cf9ddf677, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:36.435979 28885 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250113 21:10:36.437032 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:36.437201 28045 tablet_replica.cc:331] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a: stopping tablet replica
I20250113 21:10:36.437242 28045 raft_consensus.cc:2238] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 13 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:36.437283 28045 raft_consensus.cc:2267] T b4d8bcc498774b26a957a97b8986f203 P 85ea282909514f9090fe51fe85389b6a [term 13 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:36.448951 28045 tablet_server.cc:195] TabletServer@127.27.99.66:0 shutdown complete.
I20250113 21:10:36.449831 28045 tablet_server.cc:178] TabletServer@127.27.99.67:0 shutting down...
I20250113 21:10:36.544385 28928 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:36.544483 28928 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 13 FOLLOWER]: Advancing to term 14
I20250113 21:10:36.545130 28928 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.545281 28928 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 14 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
W20250113 21:10:36.545816 28798 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911): Network error: Client connection negotiation failed: client connection to 127.27.99.65:40911: connect: Connection refused (error 111)
W20250113 21:10:36.545893 28798 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939): Network error: Client connection negotiation failed: client connection to 127.27.99.66:40939: connect: Connection refused (error 111)
I20250113 21:10:36.545924 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677; no voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:36.546006 28928 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250113 21:10:36.665495 28928 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:36.665596 28928 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 14 FOLLOWER]: Advancing to term 15
I20250113 21:10:36.666386 28928 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.666536 28928 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 15 election: Requested vote from peers ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
W20250113 21:10:36.667037 28798 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer ea17415e1051464c83d233ab0fa803c3 (127.27.99.65:40911): Network error: Client connection negotiation failed: client connection to 127.27.99.65:40911: connect: Connection refused (error 111)
W20250113 21:10:36.667104 28798 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939): Network error: Client connection negotiation failed: client connection to 127.27.99.66:40939: connect: Connection refused (error 111)
I20250113 21:10:36.667162 28798 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 162d4645e5644b3aa05e447cf9ddf677; no voters: 85ea282909514f9090fe51fe85389b6a, ea17415e1051464c83d233ab0fa803c3
I20250113 21:10:36.667241 28928 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20250113 21:10:36.702244 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:36.702368 28045 tablet_replica.cc:331] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677: stopping tablet replica
I20250113 21:10:36.702422 28045 raft_consensus.cc:2238] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 15 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:36.702481 28045 raft_consensus.cc:2267] T b4d8bcc498774b26a957a97b8986f203 P 162d4645e5644b3aa05e447cf9ddf677 [term 15 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:36.714237 28045 tablet_server.cc:195] TabletServer@127.27.99.67:0 shutdown complete.
I20250113 21:10:36.715099 28045 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250113 21:10:36.715237 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:36.717368 28936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:36.718048 28939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:36.718732 28937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:36.719394 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:36.719493 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:36.719527 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:36.719547 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802636719547 us; error 0 us; skew 500 ppm
I20250113 21:10:36.720293 28045 webserver.cc:458] Webserver started at http://127.27.99.65:33161/ using document root <none> and password file <none>
I20250113 21:10:36.720373 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:36.720418 28045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 21:10:36.721060 28045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:36.722509 28946 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:36.722615 28045 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:36.722676 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root
uuid: "ea17415e1051464c83d233ab0fa803c3"
format_stamp: "Formatted at 2025-01-13 21:10:34 on dist-test-slave-4hc8"
I20250113 21:10:36.722728 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736802601520715-28045-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5267
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:36.733306 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:36.733479 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:36.733613 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:36.733881 28953 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 21:10:36.734530 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 21:10:36.734575 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 21:10:36.734596 28045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 21:10:36.734993 28045 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 21:10:36.735023 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:36.735037 28953 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Bootstrap starting.
I20250113 21:10:36.736259 28953 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 21:10:36.736495 28953 tablet_bootstrap.cc:492] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Bootstrap complete.
I20250113 21:10:36.736586 28953 ts_tablet_manager.cc:1397] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250113 21:10:36.736704 28953 raft_consensus.cc:357] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.736770 28953 raft_consensus.cc:738] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea17415e1051464c83d233ab0fa803c3, State: Initialized, Role: FOLLOWER
I20250113 21:10:36.736819 28953 consensus_queue.cc:260] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 12.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.736979 28953 ts_tablet_manager.cc:1428] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:36.738083 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.65:40911
I20250113 21:10:36.739866 29012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.65:40911 every 8 connection(s)
W20250113 21:10:36.740592 29013 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250113 21:10:36.745988 28045 tablet_server.cc:178] TabletServer@127.27.99.65:40911 shutting down...
I20250113 21:10:36.861565 28990 raft_consensus.cc:491] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:36.861660 28990 raft_consensus.cc:3054] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 12 FOLLOWER]: Advancing to term 13
I20250113 21:10:36.862406 28990 raft_consensus.cc:513] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea17415e1051464c83d233ab0fa803c3" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 40911 } } peers { permanent_uuid: "162d4645e5644b3aa05e447cf9ddf677" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 38113 } } peers { permanent_uuid: "85ea282909514f9090fe51fe85389b6a" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 40939 } }
I20250113 21:10:36.862607 28990 leader_election.cc:290] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 13 election: Requested vote from peers 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113), 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939)
W20250113 21:10:36.863255 28950 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 85ea282909514f9090fe51fe85389b6a (127.27.99.66:40939): Network error: Client connection negotiation failed: client connection to 127.27.99.66:40939: connect: Connection refused (error 111)
W20250113 21:10:36.863327 28949 leader_election.cc:336] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 162d4645e5644b3aa05e447cf9ddf677 (127.27.99.67:38113): Network error: Client connection negotiation failed: client connection to 127.27.99.67:38113: connect: Connection refused (error 111)
I20250113 21:10:36.863382 28949 leader_election.cc:304] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea17415e1051464c83d233ab0fa803c3; no voters: 162d4645e5644b3aa05e447cf9ddf677, 85ea282909514f9090fe51fe85389b6a
I20250113 21:10:36.863452 28990 raft_consensus.cc:2743] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250113 21:10:36.990414 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:36.990605 28045 tablet_replica.cc:331] stopping tablet replica
I20250113 21:10:36.990677 28045 raft_consensus.cc:2238] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 13 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:36.990748 28045 raft_consensus.cc:2267] T b4d8bcc498774b26a957a97b8986f203 P ea17415e1051464c83d233ab0fa803c3 [term 13 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:37.002537 28045 tablet_server.cc:195] TabletServer@127.27.99.65:40911 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (2925 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20250113 21:10:37.007059 28045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.99.126:37945
I20250113 21:10:37.007181 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:37.009061 29020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.009754 29023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.010375 29021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:37.011039 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:37.011130 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:37.011178 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:37.011197 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802637011197 us; error 0 us; skew 500 ppm
I20250113 21:10:37.011799 28045 webserver.cc:458] Webserver started at http://127.27.99.126:36429/ using document root <none> and password file <none>
I20250113 21:10:37.011878 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:37.011924 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:37.011971 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:37.012223 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/master-0-root/instance:
uuid: "a93872e462534fc1b148fc766df757df"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.013051 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:37.013510 29028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.013640 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.013681 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/master-0-root
uuid: "a93872e462534fc1b148fc766df757df"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.013722 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:37.019865 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:37.020033 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:37.022527 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.126:37945
I20250113 21:10:37.023058 29081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.126:37945 every 8 connection(s)
I20250113 21:10:37.023202 29082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:37.024073 29082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df: Bootstrap starting.
I20250113 21:10:37.024323 29082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:37.024693 29082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df: No bootstrap required, opened a new log
I20250113 21:10:37.024784 29082 raft_consensus.cc:357] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93872e462534fc1b148fc766df757df" member_type: VOTER }
I20250113 21:10:37.024825 29082 raft_consensus.cc:383] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:37.024838 29082 raft_consensus.cc:738] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93872e462534fc1b148fc766df757df, State: Initialized, Role: FOLLOWER
I20250113 21:10:37.024870 29082 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93872e462534fc1b148fc766df757df" member_type: VOTER }
I20250113 21:10:37.024912 29082 raft_consensus.cc:397] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 21:10:37.024933 29082 raft_consensus.cc:491] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 21:10:37.024952 29082 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:37.025349 29082 raft_consensus.cc:513] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93872e462534fc1b148fc766df757df" member_type: VOTER }
I20250113 21:10:37.025406 29082 leader_election.cc:304] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a93872e462534fc1b148fc766df757df; no voters:
I20250113 21:10:37.025480 29082 leader_election.cc:290] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 21:10:37.025550 29085 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:37.025627 29082 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [sys.catalog]: configured and running, proceeding with master startup.
I20250113 21:10:37.025669 29085 raft_consensus.cc:695] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 1 LEADER]: Becoming Leader. State: Replica: a93872e462534fc1b148fc766df757df, State: Running, Role: LEADER
I20250113 21:10:37.025717 29085 consensus_queue.cc:237] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93872e462534fc1b148fc766df757df" member_type: VOTER }
I20250113 21:10:37.025987 29087 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93872e462534fc1b148fc766df757df. Latest consensus state: current_term: 1 leader_uuid: "a93872e462534fc1b148fc766df757df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93872e462534fc1b148fc766df757df" member_type: VOTER } }
I20250113 21:10:37.025971 29086 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a93872e462534fc1b148fc766df757df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93872e462534fc1b148fc766df757df" member_type: VOTER } }
I20250113 21:10:37.026052 29087 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:37.026060 29086 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:37.026298 29090 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 21:10:37.026437 29090 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 21:10:37.026952 28045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 21:10:37.027032 29090 catalog_manager.cc:1348] Generated new cluster ID: f4d31a71863540a4ac5a8918fee36a7e
I20250113 21:10:37.027070 29090 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 21:10:37.039654 29090 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 21:10:37.040082 29090 catalog_manager.cc:1505] Loading token signing keys...
I20250113 21:10:37.044842 29090 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df: Generated new TSK 0
I20250113 21:10:37.044909 29090 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 21:10:37.058683 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:37.060868 29106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.061589 29103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.062248 29104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:37.062922 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:37.063023 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:37.063047 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:37.063060 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802637063060 us; error 0 us; skew 500 ppm
I20250113 21:10:37.063678 28045 webserver.cc:458] Webserver started at http://127.27.99.65:37583/ using document root <none> and password file <none>
I20250113 21:10:37.063748 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:37.063782 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:37.063818 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:37.064049 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root/instance:
uuid: "183fd1db29034bd7aa5100084a0ee706"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.064848 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:37.065258 29111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.065388 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.065434 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root
uuid: "183fd1db29034bd7aa5100084a0ee706"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.065474 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:37.078948 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:37.079104 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:37.079226 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:37.079403 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:37.079429 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.079448 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:37.079476 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.082113 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.65:44547
I20250113 21:10:37.082170 29174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.65:44547 every 8 connection(s)
I20250113 21:10:37.082456 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:37.084175 29179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.084811 29180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.085438 29182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:37.085698 29175 heartbeater.cc:346] Connected to a master server at 127.27.99.126:37945
I20250113 21:10:37.085750 29175 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:37.085846 29175 heartbeater.cc:510] Master 127.27.99.126:37945 requested a full tablet report, sending...
I20250113 21:10:37.086079 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:37.086122 29046 ts_manager.cc:194] Registered new tserver with Master: 183fd1db29034bd7aa5100084a0ee706 (127.27.99.65:44547)
I20250113 21:10:37.086184 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:37.086217 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:37.086243 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802637086243 us; error 0 us; skew 500 ppm
I20250113 21:10:37.086884 29046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47896
I20250113 21:10:37.086925 28045 webserver.cc:458] Webserver started at http://127.27.99.66:44601/ using document root <none> and password file <none>
I20250113 21:10:37.087085 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:37.087137 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:37.087186 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:37.087479 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-1-root/instance:
uuid: "d1d99c49b3b446d0aa0696b59ee5b05c"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.088267 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:37.088646 29187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.088757 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.088789 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-1-root
uuid: "d1d99c49b3b446d0aa0696b59ee5b05c"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.088822 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:37.104769 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:37.104935 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:37.105042 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:37.105180 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:37.105202 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.105234 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:37.105248 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.108064 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.66:36843
I20250113 21:10:37.108117 29251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.66:36843 every 8 connection(s)
I20250113 21:10:37.108395 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:37.110409 29255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.111130 29258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:37.111429 29252 heartbeater.cc:346] Connected to a master server at 127.27.99.126:37945
I20250113 21:10:37.111479 29252 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:37.111553 29252 heartbeater.cc:510] Master 127.27.99.126:37945 requested a full tablet report, sending...
W20250113 21:10:37.111733 29256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:37.111814 29046 ts_manager.cc:194] Registered new tserver with Master: d1d99c49b3b446d0aa0696b59ee5b05c (127.27.99.66:36843)
I20250113 21:10:37.112141 29046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47912
I20250113 21:10:37.112352 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:37.112447 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:37.112480 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:37.112510 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802637112509 us; error 0 us; skew 500 ppm
I20250113 21:10:37.113126 28045 webserver.cc:458] Webserver started at http://127.27.99.67:42637/ using document root <none> and password file <none>
I20250113 21:10:37.113199 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:37.113256 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:37.113307 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:37.113610 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-2-root/instance:
uuid: "a8504301c76a46f498f7a1f5be2a5035"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.114545 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:37.115024 29263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.115159 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.115200 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-2-root
uuid: "a8504301c76a46f498f7a1f5be2a5035"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.115252 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:37.143683 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:37.143960 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:37.144145 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:37.144344 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:37.144371 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.144412 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:37.144438 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.147600 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.67:37859
I20250113 21:10:37.147640 29326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.67:37859 every 8 connection(s)
I20250113 21:10:37.150480 29327 heartbeater.cc:346] Connected to a master server at 127.27.99.126:37945
I20250113 21:10:37.150545 29327 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:37.150655 29327 heartbeater.cc:510] Master 127.27.99.126:37945 requested a full tablet report, sending...
I20250113 21:10:37.150874 29046 ts_manager.cc:194] Registered new tserver with Master: a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67:37859)
I20250113 21:10:37.151118 28045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003342945s
I20250113 21:10:37.151274 29046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47924
I20250113 21:10:37.157608 28045 test_util.cc:274] Using random seed: -532966169
I20250113 21:10:37.161924 29046 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47942:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 21:10:37.162065 29046 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 21:10:37.166693 29215 tablet_service.cc:1467] Processing CreateTablet for tablet ecc4392f41a748c5b3ae9ced9e3806b2 (DEFAULT_TABLE table=test-workload [id=f781f1627a704033ac3f0deab6dc7eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:37.166814 29290 tablet_service.cc:1467] Processing CreateTablet for tablet ecc4392f41a748c5b3ae9ced9e3806b2 (DEFAULT_TABLE table=test-workload [id=f781f1627a704033ac3f0deab6dc7eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:37.166818 29215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc4392f41a748c5b3ae9ced9e3806b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:37.166908 29290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc4392f41a748c5b3ae9ced9e3806b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:37.166870 29139 tablet_service.cc:1467] Processing CreateTablet for tablet ecc4392f41a748c5b3ae9ced9e3806b2 (DEFAULT_TABLE table=test-workload [id=f781f1627a704033ac3f0deab6dc7eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:37.167047 29139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc4392f41a748c5b3ae9ced9e3806b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:37.168123 29349 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035: Bootstrap starting.
I20250113 21:10:37.168431 29350 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c: Bootstrap starting.
I20250113 21:10:37.168701 29349 tablet_bootstrap.cc:654] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:37.168849 29350 tablet_bootstrap.cc:654] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:37.169178 29351 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Bootstrap starting.
I20250113 21:10:37.169291 29349 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035: No bootstrap required, opened a new log
I20250113 21:10:37.169345 29349 ts_tablet_manager.cc:1397] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:37.169523 29349 raft_consensus.cc:357] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.169601 29349 raft_consensus.cc:383] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:37.169611 29351 tablet_bootstrap.cc:654] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:37.169637 29349 raft_consensus.cc:738] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8504301c76a46f498f7a1f5be2a5035, State: Initialized, Role: FOLLOWER
I20250113 21:10:37.169699 29349 consensus_queue.cc:260] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.169917 29327 heartbeater.cc:502] Master 127.27.99.126:37945 was elected leader, sending a full tablet report...
I20250113 21:10:37.169919 29349 ts_tablet_manager.cc:1428] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:37.170116 29350 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c: No bootstrap required, opened a new log
I20250113 21:10:37.170171 29351 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: No bootstrap required, opened a new log
I20250113 21:10:37.170176 29350 ts_tablet_manager.cc:1397] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 21:10:37.170231 29351 ts_tablet_manager.cc:1397] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:37.170367 29350 raft_consensus.cc:357] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.170370 29351 raft_consensus.cc:357] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.170454 29350 raft_consensus.cc:383] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:37.170475 29351 raft_consensus.cc:383] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:37.170482 29350 raft_consensus.cc:738] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d99c49b3b446d0aa0696b59ee5b05c, State: Initialized, Role: FOLLOWER
I20250113 21:10:37.170514 29351 raft_consensus.cc:738] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183fd1db29034bd7aa5100084a0ee706, State: Initialized, Role: FOLLOWER
I20250113 21:10:37.170573 29350 consensus_queue.cc:260] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.170603 29351 consensus_queue.cc:260] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.170770 29350 ts_tablet_manager.cc:1428] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:37.170804 29252 heartbeater.cc:502] Master 127.27.99.126:37945 was elected leader, sending a full tablet report...
I20250113 21:10:37.170855 29175 heartbeater.cc:502] Master 127.27.99.126:37945 was elected leader, sending a full tablet report...
I20250113 21:10:37.170801 29351 ts_tablet_manager.cc:1428] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:37.220781 29357 raft_consensus.cc:491] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 21:10:37.220873 29357 raft_consensus.cc:513] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.221112 29357 leader_election.cc:290] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67:37859), d1d99c49b3b446d0aa0696b59ee5b05c (127.27.99.66:36843)
I20250113 21:10:37.224448 29301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc4392f41a748c5b3ae9ced9e3806b2" candidate_uuid: "183fd1db29034bd7aa5100084a0ee706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504301c76a46f498f7a1f5be2a5035" is_pre_election: true
I20250113 21:10:37.224452 29226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc4392f41a748c5b3ae9ced9e3806b2" candidate_uuid: "183fd1db29034bd7aa5100084a0ee706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" is_pre_election: true
I20250113 21:10:37.224548 29301 raft_consensus.cc:2463] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183fd1db29034bd7aa5100084a0ee706 in term 0.
I20250113 21:10:37.224578 29226 raft_consensus.cc:2463] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183fd1db29034bd7aa5100084a0ee706 in term 0.
I20250113 21:10:37.224737 29113 leader_election.cc:304] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183fd1db29034bd7aa5100084a0ee706, a8504301c76a46f498f7a1f5be2a5035; no voters:
I20250113 21:10:37.224835 29357 raft_consensus.cc:2798] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 21:10:37.224877 29357 raft_consensus.cc:491] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 21:10:37.224892 29357 raft_consensus.cc:3054] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:37.225531 29357 raft_consensus.cc:513] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.225651 29357 leader_election.cc:290] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [CANDIDATE]: Term 1 election: Requested vote from peers a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67:37859), d1d99c49b3b446d0aa0696b59ee5b05c (127.27.99.66:36843)
I20250113 21:10:37.225806 29301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc4392f41a748c5b3ae9ced9e3806b2" candidate_uuid: "183fd1db29034bd7aa5100084a0ee706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504301c76a46f498f7a1f5be2a5035"
I20250113 21:10:37.225817 29226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc4392f41a748c5b3ae9ced9e3806b2" candidate_uuid: "183fd1db29034bd7aa5100084a0ee706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c"
I20250113 21:10:37.225898 29301 raft_consensus.cc:3054] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:37.225914 29226 raft_consensus.cc:3054] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:37.226591 29226 raft_consensus.cc:2463] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183fd1db29034bd7aa5100084a0ee706 in term 1.
I20250113 21:10:37.226656 29301 raft_consensus.cc:2463] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183fd1db29034bd7aa5100084a0ee706 in term 1.
I20250113 21:10:37.226769 29115 leader_election.cc:304] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183fd1db29034bd7aa5100084a0ee706, d1d99c49b3b446d0aa0696b59ee5b05c; no voters:
I20250113 21:10:37.226869 29357 raft_consensus.cc:2798] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:37.227041 29357 raft_consensus.cc:695] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Becoming Leader. State: Replica: 183fd1db29034bd7aa5100084a0ee706, State: Running, Role: LEADER
I20250113 21:10:37.227102 29357 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.227622 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: term changed from 0 to 1, leader changed from <none> to 183fd1db29034bd7aa5100084a0ee706 (127.27.99.65). New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.239466 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 21:10:37.239508 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 21:10:37.239724 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 21:10:37.239830 29357 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 21:10:37.257341 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.257730 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250113 21:10:37.257825 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250113 21:10:37.257942 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250113 21:10:37.258075 29357 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250113 21:10:37.258555 29359 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index -1 to 17, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.258683 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index -1 to 17, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.259529 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.17: config changed from index -1 to 17, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.259624 29046 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index -1 to 17, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.259850 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.260382 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250113 21:10:37.260538 29357 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20250113 21:10:37.260557 29044 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 17: no extra replica candidate found for tablet ecc4392f41a748c5b3ae9ced9e3806b2 (table test-workload [id=f781f1627a704033ac3f0deab6dc7eaa]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250113 21:10:37.260808 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250113 21:10:37.261009 29357 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250113 21:10:37.261155 29357 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.261298 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.261663 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.262097 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 17 to 18, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.262414 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.262710 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250113 21:10:37.262910 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250113 21:10:37.263201 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250113 21:10:37.263367 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250113 21:10:37.263862 29359 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.264003 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.264727 29046 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 18 to 19, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.264786 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.264816 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.19: config changed from index 18 to 19, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.265221 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250113 21:10:37.265386 29376 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250113 21:10:37.265472 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250113 21:10:37.265830 29359 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.266136 29357 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250113 21:10:37.266507 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.266911 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.266968 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 19 to 20, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.267024 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.267598 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250113 21:10:37.267797 29357 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250113 21:10:37.267820 29225 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250113 21:10:37.268021 29357 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250113 21:10:37.268453 29357 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.269055 29225 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.269241 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.21: config changed from index 20 to 21, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.269299 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.269942 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250113 21:10:37.269991 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 20 to 21, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER, now has a pending config: VOTER a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67), VOTER d1d99c49b3b446d0aa0696b59ee5b05c (127.27.99.66), VOTER 183fd1db29034bd7aa5100084a0ee706 (127.27.99.65). New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.270216 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250113 21:10:37.270313 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250113 21:10:37.270416 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250113 21:10:37.270819 29359 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.271010 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.271780 29046 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 21 to 22, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67), VOTER d1d99c49b3b446d0aa0696b59ee5b05c (127.27.99.66), VOTER 183fd1db29034bd7aa5100084a0ee706 (127.27.99.65). New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.272126 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.272300 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.272614 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250113 21:10:37.272724 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250113 21:10:37.272969 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250113 21:10:37.273154 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250113 21:10:37.273469 29357 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.273607 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.274456 29045 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index 22 to 23, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.274639 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.274758 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.23: config changed from index 22 to 23, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.275723 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250113 21:10:37.275836 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250113 21:10:37.275947 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250113 21:10:37.276057 29384 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250113 21:10:37.276510 29384 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.276649 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.276695 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.277570 29045 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index 23 to 24, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.277717 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.278153 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250113 21:10:37.278216 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250113 21:10:37.278409 29376 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250113 21:10:37.278570 29384 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250113 21:10:37.278885 29384 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.278988 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.279104 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.25: config changed from index 24 to 25, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.279829 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 reported cstate change: config changed from index 24 to 25, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.279877 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.280272 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250113 21:10:37.280272 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250113 21:10:37.280454 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250113 21:10:37.280515 29384 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250113 21:10:37.280962 29384 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.281111 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.281301 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.281914 29046 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index 25 to 26, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.282112 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.282408 29029 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 17: aborting the task: latest config opid_index 25; task opid_index 17
I20250113 21:10:37.282434 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250113 21:10:37.282567 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250113 21:10:37.282656 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250113 21:10:37.282963 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250113 21:10:37.283044 29357 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.283181 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.283241 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.283908 29046 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 26 to 27, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.284061 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.284372 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250113 21:10:37.284423 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250113 21:10:37.284546 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250113 21:10:37.284659 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250113 21:10:37.285117 29357 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.285272 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.285913 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.286100 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 reported cstate change: config changed from index 27 to 28, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.286477 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.286873 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250113 21:10:37.286918 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250113 21:10:37.287042 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250113 21:10:37.287114 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250113 21:10:37.287570 29384 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.287665 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.288319 29045 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index 28 to 29, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.288447 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.288494 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.288872 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250113 21:10:37.289021 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250113 21:10:37.289407 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250113 21:10:37.289608 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250113 21:10:37.289788 29357 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.289896 29031 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 19: aborting the task: latest config opid_index 29; task opid_index 19
I20250113 21:10:37.289986 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.290704 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 29 to 30, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.291127 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.291483 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250113 21:10:37.291487 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250113 21:10:37.291648 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250113 21:10:37.291707 29384 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250113 21:10:37.291958 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.292217 29384 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.292814 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.293133 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.31: config changed from index 30 to 31, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.293694 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index 30 to 31, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.293732 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.294131 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250113 21:10:37.294301 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250113 21:10:37.294498 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250113 21:10:37.294875 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250113 21:10:37.295008 29384 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.295120 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.295423 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.296041 29045 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 31 to 32, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.296041 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.296741 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250113 21:10:37.296897 29384 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250113 21:10:37.296902 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250113 21:10:37.297297 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250113 21:10:37.297405 29384 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.297552 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.297552 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.33: config changed from index 32 to 33, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.298648 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.298736 29044 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 reported cstate change: config changed from index 32 to 33, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.298986 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250113 21:10:37.299147 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250113 21:10:37.299775 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250113 21:10:37.299947 29359 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.300822 29376 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250113 21:10:37.300881 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.301131 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.301549 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.302007 29045 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index 33 to 34, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.302150 29301 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250113 21:10:37.302291 29376 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250113 21:10:37.302454 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250113 21:10:37.302655 29385 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250113 21:10:37.303071 29376 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.303212 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.304052 29046 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 reported cstate change: config changed from index 34 to 35, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:37.304077 29149 consensus_queue.cc:237] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.304052 29301 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Committing config change with OpId 1.35: config changed from index 34 to 35, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: NON_VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.304656 29226 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250113 21:10:37.304867 29359 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250113 21:10:37.305039 29300 raft_consensus.cc:1270] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 LEARNER]: Refusing update from remote peer 183fd1db29034bd7aa5100084a0ee706: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250113 21:10:37.305205 29384 consensus_queue.cc:1035] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250113 21:10:37.305428 29385 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.305509 29226 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.305755 29300 raft_consensus.cc:2949] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.306344 29046 catalog_manager.cc:5526] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c reported cstate change: config changed from index 35 to 36, a8504301c76a46f498f7a1f5be2a5035 (127.27.99.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "183fd1db29034bd7aa5100084a0ee706" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } } }
I20250113 21:10:37.306448 28045 timestamp_advancement-itest.cc:261] GCing logs...
I20250113 21:10:37.306497 28045 log_reader.cc:385] T ecc4392f41a748c5b3ae9ced9e3806b2: removed 1 log segments from log reader
I20250113 21:10:37.306522 28045 log.cc:1078] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Deleting log segment in path: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root/wals/ecc4392f41a748c5b3ae9ced9e3806b2/wal-000000001 (ops 1-16)
I20250113 21:10:37.307104 28045 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250113 21:10:37.307157 28045 master.cc:537] Master@127.27.99.126:37945 shutting down...
I20250113 21:10:37.320196 29032 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 27: aborting the task: latest config opid_index 36; task opid_index 27
I20250113 21:10:37.325382 29032 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 23: aborting the task: latest config opid_index 36; task opid_index 23
I20250113 21:10:37.333542 29032 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 35: aborting the task: latest config opid_index 36; task opid_index 35
I20250113 21:10:37.348304 29030 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 31: aborting the task: latest config opid_index 36; task opid_index 31
I20250113 21:10:37.350499 29030 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 29: aborting the task: latest config opid_index 36; task opid_index 29
I20250113 21:10:37.350644 29031 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc4392f41a748c5b3ae9ced9e3806b2 with cas_config_opid_index 33: aborting the task: latest config opid_index 36; task opid_index 33
I20250113 21:10:37.520879 28045 raft_consensus.cc:2238] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 1 LEADER]: Raft consensus shutting down.
I20250113 21:10:37.520983 28045 raft_consensus.cc:2267] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:37.521057 28045 tablet_replica.cc:331] T 00000000000000000000000000000000 P a93872e462534fc1b148fc766df757df: stopping tablet replica
I20250113 21:10:37.533504 28045 master.cc:559] Master@127.27.99.126:37945 shutdown complete.
I20250113 21:10:37.534670 28045 tablet_server.cc:178] TabletServer@127.27.99.65:0 shutting down...
I20250113 21:10:37.582947 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:37.583113 28045 tablet_replica.cc:331] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: stopping tablet replica
I20250113 21:10:37.583199 28045 raft_consensus.cc:2238] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 LEADER]: Raft consensus shutting down.
I20250113 21:10:37.583289 28045 raft_consensus.cc:2267] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:37.595860 28045 tablet_server.cc:195] TabletServer@127.27.99.65:0 shutdown complete.
I20250113 21:10:37.596839 28045 tablet_server.cc:178] TabletServer@127.27.99.66:0 shutting down...
I20250113 21:10:37.608820 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:37.608980 28045 tablet_replica.cc:331] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c: stopping tablet replica
I20250113 21:10:37.609025 28045 raft_consensus.cc:2238] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:37.609068 28045 raft_consensus.cc:2267] T ecc4392f41a748c5b3ae9ced9e3806b2 P d1d99c49b3b446d0aa0696b59ee5b05c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:37.621019 28045 tablet_server.cc:195] TabletServer@127.27.99.66:0 shutdown complete.
I20250113 21:10:37.621760 28045 tablet_server.cc:178] TabletServer@127.27.99.67:0 shutting down...
I20250113 21:10:37.648304 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:37.648494 28045 tablet_replica.cc:331] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035: stopping tablet replica
I20250113 21:10:37.648574 28045 raft_consensus.cc:2238] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:37.648628 28045 raft_consensus.cc:2267] T ecc4392f41a748c5b3ae9ced9e3806b2 P a8504301c76a46f498f7a1f5be2a5035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:37.660389 28045 tablet_server.cc:195] TabletServer@127.27.99.67:0 shutdown complete.
I20250113 21:10:37.661211 28045 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250113 21:10:37.661376 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:37.663350 29386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.664062 29389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.664748 29387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:37.665395 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:37.665477 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:37.665500 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:37.665511 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802637665512 us; error 0 us; skew 500 ppm
I20250113 21:10:37.666129 28045 webserver.cc:458] Webserver started at http://127.27.99.65:37583/ using document root <none> and password file <none>
I20250113 21:10:37.666206 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:37.666232 28045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 21:10:37.666807 28045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.667956 29396 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.668088 28045 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250113 21:10:37.668135 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root
uuid: "183fd1db29034bd7aa5100084a0ee706"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.668176 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736802601520715-28045-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5293
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:37.682179 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:37.682337 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:37.682446 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:37.683135 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 21:10:37.683182 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 21:10:37.683588 28045 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 21:10:37.683619 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.683632 29403 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Bootstrap starting.
I20250113 21:10:37.685292 29403 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 21:10:37.685510 29403 tablet_bootstrap.cc:492] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Bootstrap complete.
I20250113 21:10:37.685604 29403 ts_tablet_manager.cc:1397] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 21:10:37.685724 29403 raft_consensus.cc:357] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.685801 29403 raft_consensus.cc:738] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183fd1db29034bd7aa5100084a0ee706, State: Initialized, Role: FOLLOWER
I20250113 21:10:37.685889 29403 consensus_queue.cc:260] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "a8504301c76a46f498f7a1f5be2a5035" member_type: VOTER last_known_addr { host: "127.27.99.67" port: 37859 } } peers { permanent_uuid: "d1d99c49b3b446d0aa0696b59ee5b05c" member_type: VOTER last_known_addr { host: "127.27.99.66" port: 36843 } } peers { permanent_uuid: "183fd1db29034bd7aa5100084a0ee706" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 44547 } }
I20250113 21:10:37.686053 29403 ts_tablet_manager.cc:1428] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250113 21:10:37.686978 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.65:44547
I20250113 21:10:37.689020 29462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.65:44547 every 8 connection(s)
W20250113 21:10:37.689641 29463 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250113 21:10:37.693718 29416 tablet_service.cc:2908] Rejecting scan request for tablet ecc4392f41a748c5b3ae9ced9e3806b2: Uninitialized: clean time has not yet been initialized
I20250113 21:10:37.693945 28045 timestamp_advancement-itest.cc:441] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20250113 21:10:37.694236 28045 tablet_server.cc:178] TabletServer@127.27.99.65:44547 shutting down...
I20250113 21:10:37.939570 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:37.939805 28045 tablet_replica.cc:331] stopping tablet replica
I20250113 21:10:37.939879 28045 raft_consensus.cc:2238] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 21:10:37.939924 28045 raft_consensus.cc:2267] T ecc4392f41a748c5b3ae9ced9e3806b2 P 183fd1db29034bd7aa5100084a0ee706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:37.951673 28045 tablet_server.cc:195] TabletServer@127.27.99.65:44547 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (948 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20250113 21:10:37.955940 28045 timestamp_advancement-itest.cc:454] Running with 1 replicas
I20250113 21:10:37.956077 28045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.99.126:46487
I20250113 21:10:37.956223 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:37.958220 29467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.958959 29468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:37.959621 29470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:37.960371 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:37.960461 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:37.960489 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:37.960510 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802637960509 us; error 0 us; skew 500 ppm
I20250113 21:10:37.961190 28045 webserver.cc:458] Webserver started at http://127.27.99.126:40193/ using document root <none> and password file <none>
I20250113 21:10:37.961289 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:37.961333 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:37.961377 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:37.961623 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root/instance:
uuid: "1edc1bd5830d43ff8019d5eef338a4c3"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.962754 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:37.963302 29475 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:37.963433 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:37.963479 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root
uuid: "1edc1bd5830d43ff8019d5eef338a4c3"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:37.963528 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:37.981199 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:37.981410 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:37.983807 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.126:46487
I20250113 21:10:37.984335 29528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.126:46487 every 8 connection(s)
I20250113 21:10:37.984481 29529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:37.985414 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: Bootstrap starting.
I20250113 21:10:37.985707 29529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:37.986186 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: No bootstrap required, opened a new log
I20250113 21:10:37.986295 29529 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:37.986341 29529 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:37.986354 29529 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edc1bd5830d43ff8019d5eef338a4c3, State: Initialized, Role: FOLLOWER
I20250113 21:10:37.986405 29529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:37.986444 29529 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 21:10:37.986465 29529 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 21:10:37.986486 29529 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:37.986927 29529 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:37.986982 29529 leader_election.cc:304] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1edc1bd5830d43ff8019d5eef338a4c3; no voters:
I20250113 21:10:37.987083 29529 leader_election.cc:290] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 21:10:37.987191 29532 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:37.987222 29529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 21:10:37.987342 29532 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1edc1bd5830d43ff8019d5eef338a4c3, State: Running, Role: LEADER
I20250113 21:10:37.987396 29532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:37.988019 29534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1edc1bd5830d43ff8019d5eef338a4c3. Latest consensus state: current_term: 1 leader_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER } }
I20250113 21:10:37.988123 29534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:37.988015 29533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER } }
I20250113 21:10:37.988277 29533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:37.988554 28045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20250113 21:10:37.988682 29548 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250113 21:10:37.988723 29548 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250113 21:10:37.988787 29542 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 21:10:37.988903 29542 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 21:10:37.989387 29542 catalog_manager.cc:1348] Generated new cluster ID: 7a3c8bceb4854dfb98ac40ec985bc3b6
I20250113 21:10:37.989426 29542 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 21:10:37.997573 29542 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 21:10:37.998005 29542 catalog_manager.cc:1505] Loading token signing keys...
I20250113 21:10:38.008749 29542 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: Generated new TSK 0
I20250113 21:10:38.008831 29542 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 21:10:38.020362 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:38.022451 29550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:38.023222 28045 server_base.cc:1034] running on GCE node
W20250113 21:10:38.023885 29551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.024526 29553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:38.024681 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:38.024715 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:38.024729 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802638024728 us; error 0 us; skew 500 ppm
I20250113 21:10:38.025362 28045 webserver.cc:458] Webserver started at http://127.27.99.65:44339/ using document root <none> and password file <none>
I20250113 21:10:38.025435 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:38.025472 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:38.025506 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:38.025753 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root/instance:
uuid: "6e23886c84e746a49e7b08b6c8984e4d"
format_stamp: "Formatted at 2025-01-13 21:10:38 on dist-test-slave-4hc8"
I20250113 21:10:38.026597 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 21:10:38.026986 29558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.027134 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.027184 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root
uuid: "6e23886c84e746a49e7b08b6c8984e4d"
format_stamp: "Formatted at 2025-01-13 21:10:38 on dist-test-slave-4hc8"
I20250113 21:10:38.027220 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:38.034114 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:38.034274 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:38.034377 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:38.034512 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:38.034533 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.034559 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:38.034574 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.037012 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.65:42149
I20250113 21:10:38.037041 29621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.65:42149 every 8 connection(s)
I20250113 21:10:38.040074 29622 heartbeater.cc:346] Connected to a master server at 127.27.99.126:46487
I20250113 21:10:38.040143 29622 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:38.040216 29622 heartbeater.cc:510] Master 127.27.99.126:46487 requested a full tablet report, sending...
I20250113 21:10:38.040459 29492 ts_manager.cc:194] Registered new tserver with Master: 6e23886c84e746a49e7b08b6c8984e4d (127.27.99.65:42149)
I20250113 21:10:38.040501 28045 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003303474s
I20250113 21:10:38.041034 29492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52962
I20250113 21:10:38.047122 28045 test_util.cc:274] Using random seed: -532076654
I20250113 21:10:38.051421 29492 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52980:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 21:10:38.054947 29586 tablet_service.cc:1467] Processing CreateTablet for tablet d06f453eebdb49e7b320f86e38c2cf6c (DEFAULT_TABLE table=test-workload [id=a7c60f424c84480493301b901afa0de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 21:10:38.055047 29586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d06f453eebdb49e7b320f86e38c2cf6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 21:10:38.056439 29640 tablet_bootstrap.cc:492] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Bootstrap starting.
I20250113 21:10:38.056782 29640 tablet_bootstrap.cc:654] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Neither blocks nor log segments found. Creating new log.
I20250113 21:10:38.057297 29640 tablet_bootstrap.cc:492] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: No bootstrap required, opened a new log
I20250113 21:10:38.057336 29640 ts_tablet_manager.cc:1397] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:38.057442 29640 raft_consensus.cc:357] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.057484 29640 raft_consensus.cc:383] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 21:10:38.057497 29640 raft_consensus.cc:738] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e23886c84e746a49e7b08b6c8984e4d, State: Initialized, Role: FOLLOWER
I20250113 21:10:38.057535 29640 consensus_queue.cc:260] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.057579 29640 raft_consensus.cc:397] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 21:10:38.057598 29640 raft_consensus.cc:491] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 21:10:38.057617 29640 raft_consensus.cc:3054] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 0 FOLLOWER]: Advancing to term 1
I20250113 21:10:38.058107 29640 raft_consensus.cc:513] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.058164 29640 leader_election.cc:304] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e23886c84e746a49e7b08b6c8984e4d; no voters:
I20250113 21:10:38.058267 29640 leader_election.cc:290] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 21:10:38.058317 29642 raft_consensus.cc:2798] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Leader election won for term 1
I20250113 21:10:38.058379 29640 ts_tablet_manager.cc:1428] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:38.058466 29622 heartbeater.cc:502] Master 127.27.99.126:46487 was elected leader, sending a full tablet report...
I20250113 21:10:38.058456 29642 raft_consensus.cc:695] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 LEADER]: Becoming Leader. State: Replica: 6e23886c84e746a49e7b08b6c8984e4d, State: Running, Role: LEADER
I20250113 21:10:38.058611 29642 consensus_queue.cc:237] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.059022 29492 catalog_manager.cc:5526] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e23886c84e746a49e7b08b6c8984e4d (127.27.99.65). New cstate: current_term: 1 leader_uuid: "6e23886c84e746a49e7b08b6c8984e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:38.061764 28045 tablet_server.cc:178] TabletServer@127.27.99.65:0 shutting down...
I20250113 21:10:38.063953 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:38.064056 28045 tablet_replica.cc:331] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: stopping tablet replica
I20250113 21:10:38.064107 28045 raft_consensus.cc:2238] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 LEADER]: Raft consensus shutting down.
I20250113 21:10:38.064172 28045 raft_consensus.cc:2267] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:38.075767 28045 tablet_server.cc:195] TabletServer@127.27.99.65:0 shutdown complete.
I20250113 21:10:38.076547 28045 master.cc:537] Master@127.27.99.126:46487 shutting down...
I20250113 21:10:38.079002 28045 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 LEADER]: Raft consensus shutting down.
I20250113 21:10:38.079056 28045 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:38.079123 28045 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: stopping tablet replica
I20250113 21:10:38.090904 28045 master.cc:559] Master@127.27.99.126:46487 shutdown complete.
I20250113 21:10:38.092558 28045 timestamp_advancement-itest.cc:454] Running with 3 replicas
I20250113 21:10:38.092734 28045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.99.126:39129
I20250113 21:10:38.092864 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:38.094774 29647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.095533 29648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.096128 29650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:38.096763 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:38.096834 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:38.096860 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:38.096872 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802638096872 us; error 0 us; skew 500 ppm
I20250113 21:10:38.097553 28045 webserver.cc:458] Webserver started at http://127.27.99.126:44895/ using document root <none> and password file <none>
I20250113 21:10:38.097618 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:38.097644 28045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 21:10:38.098273 28045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:38.098706 29656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.098826 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.098870 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root
uuid: "1edc1bd5830d43ff8019d5eef338a4c3"
format_stamp: "Formatted at 2025-01-13 21:10:37 on dist-test-slave-4hc8"
I20250113 21:10:38.098922 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:38.115003 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:38.115185 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:38.117611 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.126:39129
I20250113 21:10:38.118506 29708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.126:39129 every 8 connection(s)
I20250113 21:10:38.119064 29709 sys_catalog.cc:263] Verifying existing consensus state
I20250113 21:10:38.119429 29709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: Bootstrap starting.
I20250113 21:10:38.120771 29709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 21:10:38.120998 29709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: Bootstrap complete.
I20250113 21:10:38.121191 29709 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:38.121248 29709 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edc1bd5830d43ff8019d5eef338a4c3, State: Initialized, Role: FOLLOWER
I20250113 21:10:38.121304 29709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:38.121346 29709 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 21:10:38.121371 29709 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 21:10:38.121410 29709 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:38.121929 29709 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:38.121984 29709 leader_election.cc:304] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1edc1bd5830d43ff8019d5eef338a4c3; no voters:
I20250113 21:10:38.122081 29709 leader_election.cc:290] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250113 21:10:38.122128 29712 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 21:10:38.122210 29709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 21:10:38.122243 29712 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 2 LEADER]: Becoming Leader. State: Replica: 1edc1bd5830d43ff8019d5eef338a4c3, State: Running, Role: LEADER
I20250113 21:10:38.122292 29712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER }
I20250113 21:10:38.122490 29714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1edc1bd5830d43ff8019d5eef338a4c3. Latest consensus state: current_term: 2 leader_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER } }
I20250113 21:10:38.122570 29714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:38.122495 29713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc1bd5830d43ff8019d5eef338a4c3" member_type: VOTER } }
I20250113 21:10:38.122732 29713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [sys.catalog]: This master's current role is: LEADER
I20250113 21:10:38.122840 29717 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 21:10:38.123107 29717 catalog_manager.cc:670] Loaded metadata for table test-workload [id=a7c60f424c84480493301b901afa0de2]
I20250113 21:10:38.123255 29717 tablet_loader.cc:96] loaded metadata for tablet d06f453eebdb49e7b320f86e38c2cf6c (table test-workload [id=a7c60f424c84480493301b901afa0de2])
I20250113 21:10:38.123311 29717 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 21:10:38.123405 29717 catalog_manager.cc:1260] Loaded cluster ID: 7a3c8bceb4854dfb98ac40ec985bc3b6
I20250113 21:10:38.123445 29717 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 21:10:38.123534 28045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 21:10:38.124212 29717 catalog_manager.cc:1505] Loading token signing keys...
I20250113 21:10:38.124326 29717 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3: Loaded TSK: 0
I20250113 21:10:38.124431 29717 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 21:10:38.124776 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:38.126425 29729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.127065 29732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.127664 29730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:38.128293 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:38.128396 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:38.128425 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:38.128446 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802638128446 us; error 0 us; skew 500 ppm
I20250113 21:10:38.129016 28045 webserver.cc:458] Webserver started at http://127.27.99.65:41493/ using document root <none> and password file <none>
I20250113 21:10:38.129089 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:38.129124 28045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 21:10:38.129639 28045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20250113 21:10:38.130069 29737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.130172 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.130213 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root
uuid: "6e23886c84e746a49e7b08b6c8984e4d"
format_stamp: "Formatted at 2025-01-13 21:10:38 on dist-test-slave-4hc8"
I20250113 21:10:38.130260 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:38.137785 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:38.137980 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:38.138098 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:38.138351 29744 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 21:10:38.138792 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 21:10:38.138823 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 21:10:38.138850 28045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 21:10:38.139212 28045 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 21:10:38.139238 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.139242 29744 tablet_bootstrap.cc:492] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Bootstrap starting.
I20250113 21:10:38.140358 29744 tablet_bootstrap.cc:492] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 21:10:38.140594 29744 tablet_bootstrap.cc:492] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Bootstrap complete.
I20250113 21:10:38.140689 29744 ts_tablet_manager.cc:1397] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:38.140802 29744 raft_consensus.cc:357] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.140861 29744 raft_consensus.cc:738] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e23886c84e746a49e7b08b6c8984e4d, State: Initialized, Role: FOLLOWER
I20250113 21:10:38.140908 29744 consensus_queue.cc:260] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.140954 29744 raft_consensus.cc:397] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 21:10:38.140985 29744 raft_consensus.cc:491] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 21:10:38.141010 29744 raft_consensus.cc:3054] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 1 FOLLOWER]: Advancing to term 2
I20250113 21:10:38.141706 29744 raft_consensus.cc:513] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.141781 29744 leader_election.cc:304] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e23886c84e746a49e7b08b6c8984e4d; no voters:
I20250113 21:10:38.141927 29744 leader_election.cc:290] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250113 21:10:38.141983 29798 raft_consensus.cc:2798] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 2 FOLLOWER]: Leader election won for term 2
I20250113 21:10:38.142059 29744 ts_tablet_manager.cc:1428] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:38.142093 29798 raft_consensus.cc:695] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 2 LEADER]: Becoming Leader. State: Replica: 6e23886c84e746a49e7b08b6c8984e4d, State: Running, Role: LEADER
I20250113 21:10:38.142160 29798 consensus_queue.cc:237] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } }
I20250113 21:10:38.142395 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.65:44899
I20250113 21:10:38.142513 29806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.65:44899 every 8 connection(s)
I20250113 21:10:38.142730 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:38.144237 29812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.144915 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.145514 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:38.145934 29808 heartbeater.cc:346] Connected to a master server at 127.27.99.126:39129
I20250113 21:10:38.145992 29808 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:38.146082 29808 heartbeater.cc:510] Master 127.27.99.126:39129 requested a full tablet report, sending...
I20250113 21:10:38.146109 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:38.146235 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:38.146265 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:38.146282 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802638146282 us; error 0 us; skew 500 ppm
I20250113 21:10:38.146378 29673 ts_manager.cc:194] Registered new tserver with Master: 6e23886c84e746a49e7b08b6c8984e4d (127.27.99.65:44899)
I20250113 21:10:38.146425 29673 catalog_manager.cc:5526] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "6e23886c84e746a49e7b08b6c8984e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e23886c84e746a49e7b08b6c8984e4d" member_type: VOTER last_known_addr { host: "127.27.99.65" port: 42149 } health_report { overall_health: HEALTHY } } }
I20250113 21:10:38.146924 28045 webserver.cc:458] Webserver started at http://127.27.99.66:46693/ using document root <none> and password file <none>
I20250113 21:10:38.146999 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:38.147039 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:38.147066 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:38.147274 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-1-root/instance:
uuid: "5582d4fbe651419f97a50665f95a1aaf"
format_stamp: "Formatted at 2025-01-13 21:10:38 on dist-test-slave-4hc8"
I20250113 21:10:38.147473 29673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59022
I20250113 21:10:38.147964 29808 heartbeater.cc:502] Master 127.27.99.126:39129 was elected leader, sending a full tablet report...
I20250113 21:10:38.148283 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:38.148840 29821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.148955 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.149008 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-1-root
uuid: "5582d4fbe651419f97a50665f95a1aaf"
format_stamp: "Formatted at 2025-01-13 21:10:38 on dist-test-slave-4hc8"
I20250113 21:10:38.149051 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:38.159081 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:38.159294 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:38.159452 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:38.159641 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:38.159667 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.159698 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:38.159726 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.162956 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.66:36971
I20250113 21:10:38.163004 29884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.66:36971 every 8 connection(s)
I20250113 21:10:38.163291 28045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 21:10:38.165072 29889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.165696 29888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 21:10:38.166312 29891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 21:10:38.166482 29885 heartbeater.cc:346] Connected to a master server at 127.27.99.126:39129
I20250113 21:10:38.166533 29885 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:38.166612 29885 heartbeater.cc:510] Master 127.27.99.126:39129 requested a full tablet report, sending...
I20250113 21:10:38.166803 29673 ts_manager.cc:194] Registered new tserver with Master: 5582d4fbe651419f97a50665f95a1aaf (127.27.99.66:36971)
I20250113 21:10:38.166919 28045 server_base.cc:1034] running on GCE node
I20250113 21:10:38.167029 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 21:10:38.167062 28045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 21:10:38.167095 28045 hybrid_clock.cc:648] HybridClock initialized: now 1736802638167095 us; error 0 us; skew 500 ppm
I20250113 21:10:38.167295 29673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59038
I20250113 21:10:38.167811 28045 webserver.cc:458] Webserver started at http://127.27.99.67:43209/ using document root <none> and password file <none>
I20250113 21:10:38.167903 28045 fs_manager.cc:362] Metadata directory not provided
I20250113 21:10:38.167968 28045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 21:10:38.168020 28045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 21:10:38.168265 28045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-2-root/instance:
uuid: "a5bb343c23c04b3a8699507eb4734806"
format_stamp: "Formatted at 2025-01-13 21:10:38 on dist-test-slave-4hc8"
I20250113 21:10:38.169160 28045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 21:10:38.169639 29896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.169759 28045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250113 21:10:38.169801 28045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-2-root
uuid: "a5bb343c23c04b3a8699507eb4734806"
format_stamp: "Formatted at 2025-01-13 21:10:38 on dist-test-slave-4hc8"
I20250113 21:10:38.169848 28045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDiE1Zz/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736802601520715-28045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 21:10:38.176234 28045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 21:10:38.176462 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 21:10:38.176599 28045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 21:10:38.176764 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 21:10:38.176792 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.176820 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 21:10:38.176848 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 21:10:38.179960 28045 rpc_server.cc:307] RPC server started. Bound to: 127.27.99.67:43647
I20250113 21:10:38.180011 29960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.99.67:43647 every 8 connection(s)
I20250113 21:10:38.182641 29961 heartbeater.cc:346] Connected to a master server at 127.27.99.126:39129
I20250113 21:10:38.182689 29961 heartbeater.cc:463] Registering TS with master...
I20250113 21:10:38.182775 29961 heartbeater.cc:510] Master 127.27.99.126:39129 requested a full tablet report, sending...
I20250113 21:10:38.182971 29673 ts_manager.cc:194] Registered new tserver with Master: a5bb343c23c04b3a8699507eb4734806 (127.27.99.67:43647)
I20250113 21:10:38.183303 29673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59042
I20250113 21:10:38.183449 28045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003307227s
I20250113 21:10:38.189706 28045 test_util.cc:274] Using random seed: -531934070
I20250113 21:10:38.193725 28045 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250113 21:10:38.193789 28045 tablet_server.cc:178] TabletServer@127.27.99.65:0 shutting down...
I20250113 21:10:38.195724 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:38.195958 28045 tablet_replica.cc:331] stopping tablet replica
I20250113 21:10:38.196025 28045 raft_consensus.cc:2238] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 2 LEADER]: Raft consensus shutting down.
I20250113 21:10:38.196075 28045 raft_consensus.cc:2267] T d06f453eebdb49e7b320f86e38c2cf6c P 6e23886c84e746a49e7b08b6c8984e4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:38.197683 28045 tablet_server.cc:195] TabletServer@127.27.99.65:0 shutdown complete.
I20250113 21:10:38.198530 28045 tablet_server.cc:178] TabletServer@127.27.99.66:0 shutting down...
I20250113 21:10:38.200917 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:38.212193 28045 tablet_server.cc:195] TabletServer@127.27.99.66:0 shutdown complete.
I20250113 21:10:38.212971 28045 tablet_server.cc:178] TabletServer@127.27.99.67:0 shutting down...
I20250113 21:10:38.215488 28045 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 21:10:38.226730 28045 tablet_server.cc:195] TabletServer@127.27.99.67:0 shutdown complete.
I20250113 21:10:38.227504 28045 master.cc:537] Master@127.27.99.126:39129 shutting down...
I20250113 21:10:38.229676 28045 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 2 LEADER]: Raft consensus shutting down.
I20250113 21:10:38.229741 28045 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1edc1bd5830d43ff8019d5eef338a4c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 21:10:38.229826 28045 tablet_replica.cc:331] stopping tablet replica
I20250113 21:10:38.231523 28045 master.cc:559] Master@127.27.99.126:39129 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (279 ms)
[----------] 4 tests from TimestampAdvancementITest (36710 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (36711 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20250113 21:10:38.235611 28045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5204: suppressed but not reported on 7 messages since previous log ~0 seconds ago
I20250113 21:10:38.235648 28045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:643: suppressed but not reported on 7 messages since previous log ~1 seconds ago
I20250113 21:10:38.235678 28045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 9 messages since previous log ~4 seconds ago
I20250113 21:10:38.235734 28045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:345: suppressed but not reported on 563 messages since previous log ~36 seconds ago