TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:261: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:326: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20250114 03:12:15.771472  4138 tablet_server.cc:178] TabletServer@127.4.10.129:0 shutting down...
I20250114 03:12:15.819656  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:15.819854  4138 tablet_replica.cc:331] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: stopping tablet replica
I20250114 03:12:15.819988  4138 raft_consensus.cc:2238] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 161 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:15.820058  4138 raft_consensus.cc:2267] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.833340  4138 tablet_server.cc:195] TabletServer@127.4.10.129:0 shutdown complete.
I20250114 03:12:15.834694  4138 tablet_server.cc:178] TabletServer@127.4.10.130:0 shutting down...
I20250114 03:12:15.848742  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:15.848940  4138 tablet_replica.cc:331] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: stopping tablet replica
I20250114 03:12:15.849030  4138 raft_consensus.cc:2238] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 161 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:15.849072  4138 raft_consensus.cc:2267] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.861245  4138 tablet_server.cc:195] TabletServer@127.4.10.130:0 shutdown complete.
I20250114 03:12:15.862211  4138 tablet_server.cc:178] TabletServer@127.4.10.131:0 shutting down...
W20250114 03:12:15.865794  4387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.10.129:43007: connect: Connection refused (error 111)
W20250114 03:12:15.866195  4387 consensus_peers.cc:487] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a -> Peer 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007): Couldn't send request to peer 5eea63c188224be98e6888084ea42808. Status: Network error: Client connection negotiation failed: client connection to 127.4.10.129:43007: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:12:15.869115  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:15.869215  4138 tablet_replica.cc:331] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: stopping tablet replica
I20250114 03:12:15.869278  4138 raft_consensus.cc:2238] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 161 LEADER]: Raft consensus shutting down.
I20250114 03:12:15.869346  4138 raft_consensus.cc:2267] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 161 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.881314  4138 tablet_server.cc:195] TabletServer@127.4.10.131:0 shutdown complete.
I20250114 03:12:15.882349  4138 master.cc:537] Master@127.4.10.190:45173 shutting down...
I20250114 03:12:15.994648  4138 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:12:15.994760  4138 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.994792  4138 tablet_replica.cc:331] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa: stopping tablet replica
I20250114 03:12:16.006979  4138 master.cc:559] Master@127.4.10.190:45173 shutdown complete.
I20250114 03:12:16.008780  4138 test_util.cc:181] -----------------------------------------------
I20250114 03:12:16.008826  4138 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0

[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN      ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250114 03:11:43.201246  4138 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.10.190:45173
I20250114 03:11:43.201572  4138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:11:43.201727  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:11:43.206277  4146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:11:43.206772  4144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:11:43.207455  4143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:11:43.208355  4138 server_base.cc:1034] running on GCE node
I20250114 03:11:43.208593  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:11:43.208647  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:11:43.208675  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824303208674 us; error 0 us; skew 500 ppm
I20250114 03:11:43.209998  4138 webserver.cc:458] Webserver started at http://127.4.10.190:37179/ using document root <none> and password file <none>
I20250114 03:11:43.210175  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:11:43.210237  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:11:43.210322  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:11:43.211371  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/master-0-root/instance:
uuid: "f2082d9a67064f508e2f7e78d7dd65fa"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.212651  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.002s
I20250114 03:11:43.213246  4151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.213416  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.001s
I20250114 03:11:43.213479  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/master-0-root
uuid: "f2082d9a67064f508e2f7e78d7dd65fa"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.213529  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:11:43.226241  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:11:43.226482  4138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:11:43.226573  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:11:43.226861  4157 maintenance_manager.cc:345] Maintenance manager is disabled (check --enable_maintenance_manager).
I20250114 03:11:43.230414  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.190:45173
I20250114 03:11:43.230481  4204 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.190:45173 every 8 connection(s)
I20250114 03:11:43.231015  4205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:11:43.233608  4205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa: Bootstrap starting.
I20250114 03:11:43.234402  4205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa: Neither blocks nor log segments found. Creating new log.
I20250114 03:11:43.234627  4205 log.cc:826] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa: Log is configured to *not* fsync() on all Append() calls
I20250114 03:11:43.235117  4205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa: No bootstrap required, opened a new log
I20250114 03:11:43.236408  4205 raft_consensus.cc:357] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" member_type: VOTER }
I20250114 03:11:43.236505  4205 raft_consensus.cc:383] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:11:43.236524  4205 raft_consensus.cc:738] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2082d9a67064f508e2f7e78d7dd65fa, State: Initialized, Role: FOLLOWER
I20250114 03:11:43.236613  4205 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" member_type: VOTER }
I20250114 03:11:43.236676  4205 raft_consensus.cc:397] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:11:43.236707  4205 raft_consensus.cc:491] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:11:43.236737  4205 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:11:43.237175  4205 raft_consensus.cc:513] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" member_type: VOTER }
I20250114 03:11:43.237248  4205 leader_election.cc:304] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2082d9a67064f508e2f7e78d7dd65fa; no voters: 
I20250114 03:11:43.237355  4205 leader_election.cc:290] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [CANDIDATE]: Term 1 election: Requested vote from peers 
I20250114 03:11:43.237411  4208 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:11:43.237565  4208 raft_consensus.cc:695] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 1 LEADER]: Becoming Leader. State: Replica: f2082d9a67064f508e2f7e78d7dd65fa, State: Running, Role: LEADER
I20250114 03:11:43.237569  4205 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:11:43.237653  4208 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" member_type: VOTER }
I20250114 03:11:43.237994  4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" member_type: VOTER } }
I20250114 03:11:43.238016  4210 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2082d9a67064f508e2f7e78d7dd65fa. Latest consensus state: current_term: 1 leader_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2082d9a67064f508e2f7e78d7dd65fa" member_type: VOTER } }
I20250114 03:11:43.238106  4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [sys.catalog]: This master's current role is: LEADER
I20250114 03:11:43.238179  4210 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [sys.catalog]: This master's current role is: LEADER
I20250114 03:11:43.238451  4215 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:11:43.238940  4215 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:11:43.239100  4138 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 03:11:43.240286  4215 catalog_manager.cc:1348] Generated new cluster ID: 00e0aac30ba342e8a2f88617e1000f72
I20250114 03:11:43.240334  4215 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:11:43.249203  4215 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:11:43.249684  4215 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:11:43.255239  4215 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa: Generated new TSK 0
I20250114 03:11:43.255404  4215 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:11:43.271085  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:11:43.273563  4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:11:43.274186  4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:11:43.274861  4138 server_base.cc:1034] running on GCE node
W20250114 03:11:43.275894  4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:11:43.276067  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:11:43.276121  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:11:43.276142  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824303276142 us; error 0 us; skew 500 ppm
I20250114 03:11:43.276906  4138 webserver.cc:458] Webserver started at http://127.4.10.129:32903/ using document root <none> and password file <none>
I20250114 03:11:43.277002  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:11:43.277053  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:11:43.277102  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:11:43.277418  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-0-root/instance:
uuid: "5eea63c188224be98e6888084ea42808"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.278618  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
I20250114 03:11:43.279219  4235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.279381  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:11:43.279433  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-0-root
uuid: "5eea63c188224be98e6888084ea42808"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.279482  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:11:43.298710  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:11:43.299268  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:11:43.299477  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:11:43.299798  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:11:43.299836  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.299857  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:11:43.299870  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.304183  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.129:43007
I20250114 03:11:43.304234  4298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.129:43007 every 8 connection(s)
I20250114 03:11:43.304627  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:11:43.306947  4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:11:43.307790  4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:11:43.308645  4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:11:43.309487  4299 heartbeater.cc:346] Connected to a master server at 127.4.10.190:45173
I20250114 03:11:43.309520  4138 server_base.cc:1034] running on GCE node
I20250114 03:11:43.309623  4299 heartbeater.cc:463] Registering TS with master...
I20250114 03:11:43.309768  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:11:43.309808  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:11:43.309827  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824303309827 us; error 0 us; skew 500 ppm
I20250114 03:11:43.309896  4299 heartbeater.cc:510] Master 127.4.10.190:45173 requested a full tablet report, sending...
I20250114 03:11:43.310422  4168 ts_manager.cc:194] Registered new tserver with Master: 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007)
I20250114 03:11:43.310667  4138 webserver.cc:458] Webserver started at http://127.4.10.130:39597/ using document root <none> and password file <none>
I20250114 03:11:43.310777  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:11:43.310830  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:11:43.310882  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:11:43.311187  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-1-root/instance:
uuid: "8a546775046844008af4967b5e3e1d14"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.311395  4168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51862
I20250114 03:11:43.312183  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:11:43.312665  4311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.312789  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.312839  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-1-root
uuid: "8a546775046844008af4967b5e3e1d14"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.312880  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:11:43.324621  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:11:43.324882  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:11:43.325054  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:11:43.325299  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:11:43.325336  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.325366  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:11:43.325393  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.329772  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.130:35783
I20250114 03:11:43.330272  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250114 03:11:43.331136  4374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.130:35783 every 8 connection(s)
W20250114 03:11:43.332898  4379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:11:43.333827  4378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:11:43.334151  4375 heartbeater.cc:346] Connected to a master server at 127.4.10.190:45173
I20250114 03:11:43.334223  4375 heartbeater.cc:463] Registering TS with master...
I20250114 03:11:43.334333  4375 heartbeater.cc:510] Master 127.4.10.190:45173 requested a full tablet report, sending...
I20250114 03:11:43.334585  4168 ts_manager.cc:194] Registered new tserver with Master: 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
W20250114 03:11:43.334642  4381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:11:43.335047  4168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51868
I20250114 03:11:43.335469  4138 server_base.cc:1034] running on GCE node
I20250114 03:11:43.335610  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:11:43.335639  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:11:43.335652  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824303335652 us; error 0 us; skew 500 ppm
I20250114 03:11:43.336315  4138 webserver.cc:458] Webserver started at http://127.4.10.131:38833/ using document root <none> and password file <none>
I20250114 03:11:43.336393  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:11:43.336444  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:11:43.336493  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:11:43.336779  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-2-root/instance:
uuid: "94c6eb209d3e4298b92b940281c1fd5a"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.337707  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:11:43.338225  4386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.338353  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.338398  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-2-root
uuid: "94c6eb209d3e4298b92b940281c1fd5a"
format_stamp: "Formatted at 2025-01-14 03:11:43 on dist-test-slave-ppl6"
I20250114 03:11:43.338452  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:11:43.346418  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:11:43.346665  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:11:43.346818  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:11:43.347011  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:11:43.347039  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.347071  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:11:43.347100  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:11:43.350473  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.131:39675
I20250114 03:11:43.350551  4450 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.131:39675 every 8 connection(s)
I20250114 03:11:43.357615  4451 heartbeater.cc:346] Connected to a master server at 127.4.10.190:45173
I20250114 03:11:43.357694  4451 heartbeater.cc:463] Registering TS with master...
I20250114 03:11:43.357808  4451 heartbeater.cc:510] Master 127.4.10.190:45173 requested a full tablet report, sending...
I20250114 03:11:43.358079  4168 ts_manager.cc:194] Registered new tserver with Master: 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:43.358358  4138 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00770281s
I20250114 03:11:43.358547  4168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51878
I20250114 03:11:43.367537  4138 test_util.cc:274] Using random seed: -341592724
I20250114 03:11:43.375231  4168 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51896:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20250114 03:11:43.375535  4168 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:11:43.381729  4415 tablet_service.cc:1467] Processing CreateTablet for tablet 508dc962d19543739571bcdc4c72574e (DEFAULT_TABLE table=test-workload [id=b7ec33b3d05040a991edf7dc327eaf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:11:43.381953  4415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508dc962d19543739571bcdc4c72574e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:11:43.382218  4263 tablet_service.cc:1467] Processing CreateTablet for tablet 508dc962d19543739571bcdc4c72574e (DEFAULT_TABLE table=test-workload [id=b7ec33b3d05040a991edf7dc327eaf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:11:43.382337  4263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508dc962d19543739571bcdc4c72574e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:11:43.383427  4472 tablet_bootstrap.cc:492] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Bootstrap starting.
I20250114 03:11:43.383503  4339 tablet_service.cc:1467] Processing CreateTablet for tablet 508dc962d19543739571bcdc4c72574e (DEFAULT_TABLE table=test-workload [id=b7ec33b3d05040a991edf7dc327eaf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:11:43.383613  4339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508dc962d19543739571bcdc4c72574e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:11:43.383882  4472 tablet_bootstrap.cc:654] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Neither blocks nor log segments found. Creating new log.
I20250114 03:11:43.384377  4472 tablet_bootstrap.cc:492] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: No bootstrap required, opened a new log
I20250114 03:11:43.384410  4472 ts_tablet_manager.cc:1397] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:11:43.384555  4472 raft_consensus.cc:357] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.384606  4472 raft_consensus.cc:383] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:11:43.384619  4472 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Initialized, Role: FOLLOWER
I20250114 03:11:43.384660  4472 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.384824  4472 ts_tablet_manager.cc:1428] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:11:43.384935  4299 heartbeater.cc:502] Master 127.4.10.190:45173 was elected leader, sending a full tablet report...
I20250114 03:11:43.385376  4476 tablet_bootstrap.cc:492] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Bootstrap starting.
I20250114 03:11:43.385977  4473 tablet_bootstrap.cc:492] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Bootstrap starting.
I20250114 03:11:43.386142  4476 tablet_bootstrap.cc:654] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Neither blocks nor log segments found. Creating new log.
I20250114 03:11:43.386618  4473 tablet_bootstrap.cc:654] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Neither blocks nor log segments found. Creating new log.
I20250114 03:11:43.386736  4476 tablet_bootstrap.cc:492] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: No bootstrap required, opened a new log
I20250114 03:11:43.386790  4476 ts_tablet_manager.cc:1397] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:11:43.386972  4476 raft_consensus.cc:357] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.387029  4476 raft_consensus.cc:383] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:11:43.387054  4476 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Initialized, Role: FOLLOWER
I20250114 03:11:43.387104  4476 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.387221  4473 tablet_bootstrap.cc:492] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: No bootstrap required, opened a new log
I20250114 03:11:43.387274  4473 ts_tablet_manager.cc:1397] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:11:43.387272  4476 ts_tablet_manager.cc:1428] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:11:43.387390  4375 heartbeater.cc:502] Master 127.4.10.190:45173 was elected leader, sending a full tablet report...
I20250114 03:11:43.387521  4473 raft_consensus.cc:357] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.387605  4473 raft_consensus.cc:383] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:11:43.387632  4473 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Initialized, Role: FOLLOWER
I20250114 03:11:43.387693  4473 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.387852  4473 ts_tablet_manager.cc:1428] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20250114 03:11:43.387923  4451 heartbeater.cc:502] Master 127.4.10.190:45173 was elected leader, sending a full tablet report...
I20250114 03:11:43.429679  4479 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:11:43.429788  4479 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.430135  4479 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:43.433157  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a" is_pre_election: true
I20250114 03:11:43.433157  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808" is_pre_election: true
I20250114 03:11:43.433313  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 0.
I20250114 03:11:43.433313  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 0.
I20250114 03:11:43.433499  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 
I20250114 03:11:43.433635  4479 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:11:43.433681  4479 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:43.433703  4479 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:11:43.434373  4479 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.434502  4479 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 1 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:43.434667  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:43.434667  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:43.434767  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:11:43.434782  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:11:43.435441  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 1.
I20250114 03:11:43.435441  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 1.
I20250114 03:11:43.435643  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 
I20250114 03:11:43.435729  4479 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:11:43.435890  4479 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 1 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:43.435957  4479 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.436648  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 1 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:43.456293  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 1 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:11:43.456452  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 1 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:11:43.456565  4482 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:11:43.456665  4479 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:11:43.471242  4138 test_util.cc:274] Using random seed: -341489017
I20250114 03:11:43.476676  4138 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250114 03:11:43.705335  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:43.705440  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:11:43.706235  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.706430  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 2 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:43.709777  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:43.709786  4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:43.710101  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:43.710206  4475 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 03:11:43.715688  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 2 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 1. Current term is 2. Ops: []
I20250114 03:11:43.715961  4508 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 1.39, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
W20250114 03:11:43.716099  4508 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:11:43.716131  4508 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250114 03:11:43.779299  4508 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:11:43.779417  4508 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:43.779524  4508 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.779692  4508 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:11:43.780604  4480 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:43.780671  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:11:43.781261  4480 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.781469  4480 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 2 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:43.784036  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:43.784127  4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:43.784135  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 2: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:43.784793  4349 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 2.
I20250114 03:11:43.784982  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:43.785079  4480 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:11:43.785188  4480 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 2 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:43.785251  4480 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.785771  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 1 to 2, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 2 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:43.829567  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:43.829686  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:11:43.830605  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.830834  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 3 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:43.831017  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 3 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:43.831106  4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 3 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:43.831195  4349 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:11:43.831996  4349 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 3.
I20250114 03:11:43.832239  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:43.832346  4475 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 03:11:43.832484  4475 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 3 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:43.832546  4475 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.833073  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 2 to 3, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 3 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:43.868196  4349 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 3 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 2. Current term is 3. Ops: []
I20250114 03:11:43.868499  4480 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250114 03:11:43.891039  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 3 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 2. Current term is 3. Ops: []
I20250114 03:11:43.891338  4522 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250114 03:11:43.891444  4522 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 3.40 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 39
I20250114 03:11:43.906706  4480 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 2 LEADER]: Stepping down as leader of term 2
I20250114 03:11:43.906805  4480 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:43.906881  4480 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.40, Last appended by leader: 40, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:43.906988  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:11:43.916397  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 3 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 3 index: 40. (term mismatch)
I20250114 03:11:43.916481  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 39
I20250114 03:11:43.916538  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 2.40
I20250114 03:11:43.916580  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:43.916800  4475 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250114 03:11:43.942468  4349 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 3 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20250114 03:11:43.942739  4475 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250114 03:11:44.044157  4508 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:44.044258  4508 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:11:44.044965  4508 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.045156  4508 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 4 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:44.045384  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:44.045384  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:44.045723  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:44.045892  4508 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 03:11:44.063592  4349 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 4 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 3. Current term is 4. Ops: []
I20250114 03:11:44.063876  4524 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250114 03:11:44.136440  4480 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:44.136567  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:11:44.137256  4480 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.137436  4480 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 4 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:44.137679  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:44.137681  4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:44.137948  4349 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 4: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:44.138170  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:44.138264  4480 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 03:11:44.140053  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 4 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 3. Current term is 4. Ops: []
I20250114 03:11:44.140267  4523 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250114 03:11:44.164814  4349 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 4 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 3. Current term is 4. Ops: []
I20250114 03:11:44.165103  4475 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250114 03:11:44.187244  4508 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:44.187348  4508 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 4 FOLLOWER]: Advancing to term 5
I20250114 03:11:44.188033  4508 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.188216  4508 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 5 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:44.188450  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 5 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:44.188450  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 5 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:44.188561  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 4 FOLLOWER]: Advancing to term 5
I20250114 03:11:44.189174  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 5.
I20250114 03:11:44.189333  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:44.189419  4508 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 03:11:44.189478  4508 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 5 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:44.189539  4508 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.190179  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 3 to 5, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 5 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:44.238478  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 5 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 3. Current term is 5. Ops: []
I20250114 03:11:44.238760  4523 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250114 03:11:44.258419  4349 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 5 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 3. Current term is 5. Ops: []
I20250114 03:11:44.258692  4523 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250114 03:11:44.258797  4523 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 5.41 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 40
I20250114 03:11:44.293255  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 5 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
I20250114 03:11:44.293550  4508 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250114 03:11:44.294364  4524 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 03:11:44.294435  4524 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:44.294507  4524 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.294643  4524 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:11:44.295468  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 4 FOLLOWER]: Advancing to term 5
I20250114 03:11:44.296145  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 5 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
I20250114 03:11:44.296370  4508 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250114 03:11:44.481256  4480 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:44.481354  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:11:44.482071  4480 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.482223  4480 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 6 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:44.482488  4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:44.482492  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:44.482784  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:44.482887  4480 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 03:11:44.492319  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 6 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 5. Current term is 6. Ops: []
I20250114 03:11:44.492584  4482 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 5.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250114 03:11:44.520816  4482 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 03:11:44.520920  4482 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:44.520996  4482 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 5.41, Last appended by leader: 41, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.521126  4482 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:11:44.603400  4480 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:44.603506  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:11:44.604224  4480 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.604403  4480 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 7 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:44.604574  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:44.604629  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:11:44.604645  4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 7 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:44.604645  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 7 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:44.604765  4349 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:11:44.605252  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.605355  4349 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 7.
I20250114 03:11:44.605371  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:11:44.605507  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 
I20250114 03:11:44.605536  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:44.605590  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:44.605630  4480 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 7 FOLLOWER]: Leader election won for term 7
I20250114 03:11:44.605376  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 6 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:44.605602  4347 raft_consensus.cc:2363] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 for earlier term 6. Current term is 7.
I20250114 03:11:44.605728  4480 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 7 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:44.605775  4480 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 5.41, Last appended by leader: 41, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:44.605826  4239 leader_election.cc:400] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 6 election: Vote denied by peer 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783) with higher term. Message: Invalid argument: T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 for earlier term 6. Current term is 7.
I20250114 03:11:44.605849  4239 leader_election.cc:403] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250114 03:11:44.606127  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 7.
I20250114 03:11:44.606189  4475 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783) with higher term. Message: Invalid argument: T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 for earlier term 6. Current term is 7.
I20250114 03:11:44.606545  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 5 to 7, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 7 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:44.714089  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 7 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 5 index: 41. Preceding OpId from leader: term: 7 index: 42. (index mismatch)
I20250114 03:11:44.714423  4480 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250114 03:11:44.724277  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 7 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 5 index: 41. Preceding OpId from leader: term: 7 index: 42. (index mismatch)
I20250114 03:11:44.724593  4526 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250114 03:11:45.015481  4482 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:45.015583  4482 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:11:45.016253  4482 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.016441  4482 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 8 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:45.016645  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:45.016654  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:45.016888  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:45.016976  4482 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 03:11:45.021283  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 8 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 7. Current term is 8. Ops: []
I20250114 03:11:45.021498  4527 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 7.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250114 03:11:45.055761  4480 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 7 LEADER]: Stepping down as leader of term 7
I20250114 03:11:45.055871  4480 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:45.055936  4480 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 7.42, Last appended by leader: 42, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.056062  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:11:45.073523  4524 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:45.073622  4524 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:11:45.074338  4524 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.074522  4524 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 8 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:45.074761  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:45.074784  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:45.074889  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 8: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:45.075428  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 8.
I20250114 03:11:45.075596  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:45.075706  4524 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 03:11:45.075762  4524 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 8 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:45.075819  4524 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 7.42, Last appended by leader: 42, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.076221  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 7 to 8, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 8 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:45.129415  4482 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:45.129532  4482 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 8 FOLLOWER]: Advancing to term 9
I20250114 03:11:45.130256  4482 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.130416  4482 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 9 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:45.130657  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 9 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:45.130663  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 9 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:45.130801  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 8 FOLLOWER]: Advancing to term 9
I20250114 03:11:45.131397  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 9.
I20250114 03:11:45.131624  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:45.131727  4482 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 9 FOLLOWER]: Leader election won for term 9
I20250114 03:11:45.131860  4482 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 9 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:45.131929  4482 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 7.42, Last appended by leader: 42, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.132390  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 8 to 9, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 9 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:45.177541  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 9 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 8. Current term is 9. Ops: []
I20250114 03:11:45.177889  4524 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250114 03:11:45.178009  4524 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 9.43 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20250114 03:11:45.184415  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 9 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 8. Current term is 9. Ops: []
I20250114 03:11:45.184662  4524 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250114 03:11:45.208225  4273 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 8 LEADER]: Stepping down as leader of term 8
I20250114 03:11:45.208300  4273 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:45.208352  4273 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 8.43, Last appended by leader: 43, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.208464  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 8 FOLLOWER]: Advancing to term 9
I20250114 03:11:45.209093  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 9 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 8 index: 43. Preceding OpId from leader: term: 9 index: 43. (term mismatch)
I20250114 03:11:45.209136  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 42
I20250114 03:11:45.209157  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 8.43
I20250114 03:11:45.209178  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:45.209450  4482 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250114 03:11:45.232842  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 9 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 7 index: 42. Preceding OpId from leader: term: 9 index: 43. (index mismatch)
I20250114 03:11:45.233112  4482 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250114 03:11:45.510830  4480 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:45.510946  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 9 FOLLOWER]: Advancing to term 10
I20250114 03:11:45.511603  4480 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.511734  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 10 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 9. Current term is 10. Ops: []
I20250114 03:11:45.511760  4480 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 10 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:45.511914  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:45.511950  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:45.511994  4529 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 9.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:11:45.512141  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:45.512230  4480 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250114 03:11:45.573190  4529 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 9 LEADER]: Stepping down as leader of term 9
I20250114 03:11:45.573283  4529 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:45.573362  4529 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 9.43, Last appended by leader: 43, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.573472  4529 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 9 FOLLOWER]: Advancing to term 10
I20250114 03:11:45.595860  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:45.595963  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 9 FOLLOWER]: Advancing to term 10
I20250114 03:11:45.596617  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.596793  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 10 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:45.597010  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:45.597010  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:45.597131  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 10: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:45.597704  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 10.
I20250114 03:11:45.597915  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:45.598012  4475 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 10 FOLLOWER]: Leader election won for term 10
I20250114 03:11:45.598076  4475 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 10 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:45.598133  4475 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 9.43, Last appended by leader: 43, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.598567  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 9 to 10, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 10 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:45.619601  4480 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:45.619688  4480 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 10 FOLLOWER]: Advancing to term 11
I20250114 03:11:45.620290  4480 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.620451  4480 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 11 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:45.620690  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 11 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:45.620690  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 11 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:45.620800  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 10 FOLLOWER]: Advancing to term 11
I20250114 03:11:45.621371  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 11.
I20250114 03:11:45.621541  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:45.621645  4480 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 11 FOLLOWER]: Leader election won for term 11
I20250114 03:11:45.621709  4480 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 11 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:45.621776  4480 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 9.43, Last appended by leader: 43, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.671588  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 10 to 11, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 11 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:45.686676  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 11 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 10. Current term is 11. Ops: []
I20250114 03:11:45.686976  4475 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:11:45.687851  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 11 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 10. Current term is 11. Ops: []
I20250114 03:11:45.688066  4524 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:11:45.688131  4524 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 11.44 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 43
I20250114 03:11:45.698688  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 11 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 9 index: 43. Preceding OpId from leader: term: 11 index: 44. (index mismatch)
I20250114 03:11:45.698956  4480 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:11:45.731216  4475 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 10 LEADER]: Stepping down as leader of term 10
I20250114 03:11:45.731309  4475 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:45.731386  4475 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 10.44, Last appended by leader: 44, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.731506  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 10 FOLLOWER]: Advancing to term 11
I20250114 03:11:45.732379  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.733615  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.735777  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.739940  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.743835  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 11 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 10 index: 44. Preceding OpId from leader: term: 11 index: 44. (term mismatch)
I20250114 03:11:45.743894  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 43
I20250114 03:11:45.743908  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 10.44
I20250114 03:11:45.743923  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:45.744165  4480 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:11:45.748133  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.764334  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.796563  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.860765  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.876461  4529 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:45.876564  4529 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 11 FOLLOWER]: Advancing to term 12
I20250114 03:11:45.877230  4529 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.877399  4529 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 12 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:45.877635  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:45.877642  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:45.877957  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:45.878046  4529 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250114 03:11:45.885386  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 12 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 11. Current term is 12. Ops: []
I20250114 03:11:45.885669  4527 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 11.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 03:11:45.885797  4527 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 11 LEADER]: Stepping down as leader of term 11
I20250114 03:11:45.885830  4527 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:45.885910  4527 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 11.44, Last appended by leader: 44, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.885995  4527 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 11 FOLLOWER]: Advancing to term 12
I20250114 03:11:45.957257  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:45.957361  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 11 FOLLOWER]: Advancing to term 12
I20250114 03:11:45.958099  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.958274  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 12 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:45.958483  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:45.958511  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:45.958586  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 12: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:45.959088  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 12.
I20250114 03:11:45.959272  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:45.959367  4475 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 12 FOLLOWER]: Leader election won for term 12
I20250114 03:11:45.959419  4475 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 12 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:45.959458  4475 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 11.44, Last appended by leader: 44, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.959923  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 11 to 12, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 12 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:45.988436  4529 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:45.988538  4529 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 12 FOLLOWER]: Advancing to term 13
I20250114 03:11:45.988987  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:45.989261  4529 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.989406  4529 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 13 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:45.989624  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 13 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:45.989691  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 12 FOLLOWER]: Advancing to term 13
I20250114 03:11:45.989629  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 13 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:45.990288  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 13.
I20250114 03:11:45.990468  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:45.990547  4529 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 13 FOLLOWER]: Leader election won for term 13
I20250114 03:11:45.990600  4529 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 13 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:45.990661  4529 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 11.44, Last appended by leader: 44, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:45.991122  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 12 to 13, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 13 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:46.035295  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 13 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 12. Current term is 13. Ops: []
I20250114 03:11:46.035600  4475 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 03:11:46.035722  4475 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 13.45 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 44
I20250114 03:11:46.035776  4524 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 12 LEADER]: Stepping down as leader of term 12
I20250114 03:11:46.035830  4524 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:46.035887  4524 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 12.45, Last appended by leader: 45, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.035967  4524 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 12 FOLLOWER]: Advancing to term 13
I20250114 03:11:46.068604  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 13 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 12 index: 45. Preceding OpId from leader: term: 13 index: 45. (term mismatch)
I20250114 03:11:46.068686  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 44
I20250114 03:11:46.068709  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 12.45
I20250114 03:11:46.068732  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:46.068958  4529 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 03:11:46.069942  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 13 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 11 index: 44. Preceding OpId from leader: term: 13 index: 45. (index mismatch)
I20250114 03:11:46.070228  4529 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250114 03:11:46.245210  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:46.268065  4531 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:46.268168  4531 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 13 FOLLOWER]: Advancing to term 14
I20250114 03:11:46.268893  4531 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.269048  4531 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 14 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:46.269285  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:46.269297  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:46.269520  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:46.269615  4531 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250114 03:11:46.280074  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 14 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 13. Current term is 14. Ops: []
I20250114 03:11:46.280341  4532 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 13.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 03:11:46.280470  4532 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 13 LEADER]: Stepping down as leader of term 13
I20250114 03:11:46.280501  4532 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:46.280568  4532 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 13.45, Last appended by leader: 45, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.280663  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 13 FOLLOWER]: Advancing to term 14
I20250114 03:11:46.363147  4524 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:46.363246  4524 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 13 FOLLOWER]: Advancing to term 14
I20250114 03:11:46.363930  4524 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.364097  4524 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 14 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:46.364374  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:46.364374  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:46.364506  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 14: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:46.365034  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 14.
I20250114 03:11:46.365207  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:46.365291  4524 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 14 FOLLOWER]: Leader election won for term 14
I20250114 03:11:46.365345  4524 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 14 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:46.365384  4524 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 13.45, Last appended by leader: 45, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.365852  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 13 to 14, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 14 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:46.398841  4531 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:46.398936  4531 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 14 FOLLOWER]: Advancing to term 15
I20250114 03:11:46.399657  4531 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.399837  4531 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 15 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:46.400096  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 15 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:46.400096  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 15 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:46.400206  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 14 FOLLOWER]: Advancing to term 15
I20250114 03:11:46.400779  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 15.
I20250114 03:11:46.400992  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:46.401125  4531 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 15 FOLLOWER]: Leader election won for term 15
I20250114 03:11:46.401206  4531 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 15 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:46.401255  4531 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 13.45, Last appended by leader: 45, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.401803  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 14 to 15, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 15 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:46.443195  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 15 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 14. Current term is 15. Ops: []
I20250114 03:11:46.443467  4524 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 03:11:46.443570  4524 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 15.46 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 45
I20250114 03:11:46.443609  4475 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 14 LEADER]: Stepping down as leader of term 14
I20250114 03:11:46.443677  4475 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:46.443747  4475 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 14.46, Last appended by leader: 46, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.443840  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 14 FOLLOWER]: Advancing to term 15
I20250114 03:11:46.480479  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 15 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 14 index: 46. Preceding OpId from leader: term: 15 index: 46. (term mismatch)
I20250114 03:11:46.480577  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 45
I20250114 03:11:46.480594  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 14.46
I20250114 03:11:46.480613  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:46.480866  4527 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 03:11:46.482035  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 15 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 13 index: 45. Preceding OpId from leader: term: 15 index: 46. (index mismatch)
I20250114 03:11:46.482357  4533 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250114 03:11:46.669983  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:46.670091  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 15 FOLLOWER]: Advancing to term 16
I20250114 03:11:46.670769  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.671052  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 16 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:46.671167  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:46.671190  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:46.671479  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:46.671625  4532 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250114 03:11:46.672533  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:46.672567  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 15 FOLLOWER]: Advancing to term 16
I20250114 03:11:46.673127  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.673262  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 16 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:46.673440  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:46.673461  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:46.673545  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 16: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:46.673722  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:46.673817  4475 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250114 03:11:46.693984  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 16 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 15. Current term is 16. Ops: []
I20250114 03:11:46.693984  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 16 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 15. Current term is 16. Ops: []
I20250114 03:11:46.694233  4531 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 15.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250114 03:11:46.694353  4533 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 15 LEADER]: Stepping down as leader of term 15
I20250114 03:11:46.694386  4533 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:46.694377  4527 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 15.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250114 03:11:46.694553  4533 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 15.46, Last appended by leader: 46, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.694657  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 15 FOLLOWER]: Advancing to term 16
I20250114 03:11:46.757499  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:46.809408  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:46.809526  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 16 FOLLOWER]: Advancing to term 17
I20250114 03:11:46.811548  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:46.811617  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 16 FOLLOWER]: Advancing to term 17
I20250114 03:11:46.813179  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.813427  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 17 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:46.813621  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 17 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:46.813628  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 17 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:46.813740  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 16 FOLLOWER]: Advancing to term 17
I20250114 03:11:46.813731  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.813875  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 17: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:46.813918  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 17 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:46.814406  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 17 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:46.814544  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 17 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:46.814654  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 17: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:46.814756  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:46.814872  4532 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20250114 03:11:46.814924  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 17.
I20250114 03:11:46.815071  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:46.815160  4475 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 17 FOLLOWER]: Leader election won for term 17
I20250114 03:11:46.815212  4475 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 17 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:46.815258  4475 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 15.46, Last appended by leader: 46, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:46.815694  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 15 to 17, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 17 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:46.894248  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 17 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 15 index: 46. Preceding OpId from leader: term: 17 index: 47. (index mismatch)
I20250114 03:11:46.894560  4475 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250114 03:11:46.895512  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 17 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 15 index: 46. Preceding OpId from leader: term: 17 index: 47. (index mismatch)
I20250114 03:11:46.895807  4475 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250114 03:11:47.083649  4531 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:47.083768  4531 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 17 FOLLOWER]: Advancing to term 18
I20250114 03:11:47.084477  4531 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.084646  4531 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 18 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:47.084895  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 18 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:47.084909  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 18 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:47.085139  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:47.085223  4531 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20250114 03:11:47.087999  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 18 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 17. Current term is 18. Ops: []
I20250114 03:11:47.088213  4475 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 17.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 03:11:47.088308  4475 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 17 LEADER]: Stepping down as leader of term 17
I20250114 03:11:47.088342  4475 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:47.088398  4475 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 17.47, Last appended by leader: 47, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.088497  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 17 FOLLOWER]: Advancing to term 18
I20250114 03:11:47.172813  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:47.172912  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 17 FOLLOWER]: Advancing to term 18
I20250114 03:11:47.173576  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.173753  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 18 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:47.173975  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 18 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:47.173987  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 18 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:47.174108  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 18: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:47.174583  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 18.
I20250114 03:11:47.174758  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:47.174837  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 18 FOLLOWER]: Leader election won for term 18
I20250114 03:11:47.174961  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 18 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:47.175002  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 17.47, Last appended by leader: 47, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.175462  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 17 to 18, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 18 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:47.192616  4531 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:47.192739  4531 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 18 FOLLOWER]: Advancing to term 19
I20250114 03:11:47.193426  4531 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.193591  4531 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 19 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:47.193797  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 19 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:47.193797  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 19 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:47.193922  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 18 FOLLOWER]: Advancing to term 19
I20250114 03:11:47.194478  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 19.
I20250114 03:11:47.194685  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:47.194820  4531 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 19 FOLLOWER]: Leader election won for term 19
I20250114 03:11:47.194901  4531 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 19 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:47.194976  4531 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 17.47, Last appended by leader: 47, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.195489  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 18 to 19, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 19 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:47.261049  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 19 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 18. Current term is 19. Ops: []
I20250114 03:11:47.261379  4532 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 03:11:47.261541  4532 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 19.48 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 47
I20250114 03:11:47.261596  4534 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 18 LEADER]: Stepping down as leader of term 18
I20250114 03:11:47.261660  4534 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:47.261756  4534 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 18.48, Last appended by leader: 48, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.261847  4534 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 18 FOLLOWER]: Advancing to term 19
I20250114 03:11:47.279947  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 19 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 18 index: 48. Preceding OpId from leader: term: 19 index: 48. (term mismatch)
I20250114 03:11:47.280025  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 47
I20250114 03:11:47.280041  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 18.48
I20250114 03:11:47.280057  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:47.280241  4531 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 03:11:47.281251  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 19 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 17 index: 47. Preceding OpId from leader: term: 19 index: 48. (index mismatch)
I20250114 03:11:47.281490  4531 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250114 03:11:47.461313  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:47.461414  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 19 FOLLOWER]: Advancing to term 20
I20250114 03:11:47.462116  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.462265  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 20 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:47.462481  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 20 candidate_status { last_received { term: 19 index: 48 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:47.462488  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 20 candidate_status { last_received { term: 19 index: 48 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:47.462742  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:47.462857  4475 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20250114 03:11:47.467844  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 20 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 19. Current term is 20. Ops: []
I20250114 03:11:47.468068  4533 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 19.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250114 03:11:47.468183  4533 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 19 LEADER]: Stepping down as leader of term 19
I20250114 03:11:47.468218  4533 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:47.468266  4533 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 19.48, Last appended by leader: 48, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.468348  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 19 FOLLOWER]: Advancing to term 20
I20250114 03:11:47.564512  4534 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:47.564632  4534 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 19 FOLLOWER]: Advancing to term 20
I20250114 03:11:47.565280  4534 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.565444  4534 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 20 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:47.565675  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 20 candidate_status { last_received { term: 19 index: 48 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:47.565687  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 20 candidate_status { last_received { term: 19 index: 48 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:47.565765  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 20: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:47.566351  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 20.
I20250114 03:11:47.566542  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:47.566648  4534 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 20 FOLLOWER]: Leader election won for term 20
I20250114 03:11:47.566712  4534 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 20 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:47.566771  4534 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 19.48, Last appended by leader: 48, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.567227  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 19 to 20, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 20 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:47.600329  4475 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:47.600471  4475 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 20 FOLLOWER]: Advancing to term 21
I20250114 03:11:47.601161  4475 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.601332  4475 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 21 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:47.601553  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 21 candidate_status { last_received { term: 19 index: 48 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:47.601560  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 21 candidate_status { last_received { term: 19 index: 48 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:47.601668  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 20 FOLLOWER]: Advancing to term 21
I20250114 03:11:47.602272  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 21.
I20250114 03:11:47.602454  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:47.602550  4475 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 21 FOLLOWER]: Leader election won for term 21
I20250114 03:11:47.602648  4475 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 21 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:47.602711  4475 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 19.48, Last appended by leader: 48, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.603137  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 20 to 21, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 21 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:47.665685  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 21 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 20. Current term is 21. Ops: []
I20250114 03:11:47.666014  4534 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250114 03:11:47.666133  4534 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 21.49 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 48
I20250114 03:11:47.666173  4532 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 20 LEADER]: Stepping down as leader of term 20
I20250114 03:11:47.666234  4532 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:47.666292  4532 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 20.49, Last appended by leader: 49, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.666370  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 20 FOLLOWER]: Advancing to term 21
I20250114 03:11:47.694684  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 21 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 19 index: 48. Preceding OpId from leader: term: 21 index: 49. (index mismatch)
I20250114 03:11:47.694990  4475 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250114 03:11:47.696053  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 21 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 20 index: 49. Preceding OpId from leader: term: 21 index: 49. (term mismatch)
I20250114 03:11:47.696115  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 48
I20250114 03:11:47.696138  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 20.49
I20250114 03:11:47.696161  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:47.696311  4475 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250114 03:11:47.757812  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:47.960017  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:47.960117  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 21 FOLLOWER]: Advancing to term 22
I20250114 03:11:47.960798  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.960966  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 22 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:47.961189  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 22 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:47.961189  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 22 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:47.961455  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:47.961594  4533 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20250114 03:11:47.962643  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 22 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 21. Current term is 22. Ops: []
I20250114 03:11:47.962849  4536 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 21.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250114 03:11:47.962954  4536 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 21 LEADER]: Stepping down as leader of term 21
I20250114 03:11:47.962986  4536 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:47.963047  4536 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 21.49, Last appended by leader: 49, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:47.963126  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 21 FOLLOWER]: Advancing to term 22
I20250114 03:11:48.007140  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:48.007243  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 21 FOLLOWER]: Advancing to term 22
I20250114 03:11:48.007923  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.008096  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 22 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:48.008306  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 22 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:48.008309  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 22 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:48.008456  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 22: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:48.008924  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 22.
I20250114 03:11:48.009093  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:48.009174  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 22 FOLLOWER]: Leader election won for term 22
I20250114 03:11:48.009217  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 22 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:48.009254  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 21.49, Last appended by leader: 49, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.009750  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 21 to 22, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 22 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:48.104012  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:48.104111  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 22 FOLLOWER]: Advancing to term 23
I20250114 03:11:48.104789  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.104959  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 23 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:48.105170  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 23 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:48.105242  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 22 FOLLOWER]: Advancing to term 23
I20250114 03:11:48.105175  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 23 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:48.105971  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 23.
I20250114 03:11:48.106184  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:48.106282  4533 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 23 FOLLOWER]: Leader election won for term 23
I20250114 03:11:48.106380  4533 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 23 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:48.106422  4533 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 21.49, Last appended by leader: 49, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.106946  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 22 to 23, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 23 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:48.107965  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 23 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 22. Current term is 23. Ops: []
I20250114 03:11:48.108042  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 23 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 22. Current term is 23. Ops: []
I20250114 03:11:48.108145  4534 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250114 03:11:48.108304  4532 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250114 03:11:48.108379  4532 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 23.50 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 49
I20250114 03:11:48.108429  4534 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 22 LEADER]: Stepping down as leader of term 22
I20250114 03:11:48.108465  4534 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:48.108541  4534 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 22.50, Last appended by leader: 50, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.108611  4534 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 22 FOLLOWER]: Advancing to term 23
I20250114 03:11:48.184109  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 23 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 21 index: 49. Preceding OpId from leader: term: 23 index: 50. (index mismatch)
I20250114 03:11:48.184388  4533 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250114 03:11:48.185441  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 23 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 22 index: 50. Preceding OpId from leader: term: 23 index: 50. (term mismatch)
I20250114 03:11:48.185496  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 49
I20250114 03:11:48.185537  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 22.50
I20250114 03:11:48.185559  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:48.185725  4537 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250114 03:11:48.369109  4536 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:48.369210  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 23 FOLLOWER]: Advancing to term 24
I20250114 03:11:48.369930  4536 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.370105  4536 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 24 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:48.370316  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 24 candidate_status { last_received { term: 23 index: 50 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:48.370327  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 24 candidate_status { last_received { term: 23 index: 50 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:48.370565  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:48.370666  4536 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20250114 03:11:48.378923  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 24 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 23. Current term is 24. Ops: []
I20250114 03:11:48.379249  4539 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 23.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250114 03:11:48.379403  4533 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 23 LEADER]: Stepping down as leader of term 23
I20250114 03:11:48.379456  4533 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:48.379515  4533 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 23.50, Last appended by leader: 50, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.379598  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 23 FOLLOWER]: Advancing to term 24
I20250114 03:11:48.410254  4534 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:48.410344  4534 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 23 FOLLOWER]: Advancing to term 24
I20250114 03:11:48.411026  4534 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.411202  4534 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 24 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:48.411422  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 24 candidate_status { last_received { term: 23 index: 50 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:48.411424  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 24 candidate_status { last_received { term: 23 index: 50 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:48.411581  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 24: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:48.412127  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 24.
I20250114 03:11:48.412297  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:48.412384  4534 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 24 FOLLOWER]: Leader election won for term 24
I20250114 03:11:48.412446  4534 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 24 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:48.412611  4534 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 23.50, Last appended by leader: 50, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.413054  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 23 to 24, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 24 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:48.502622  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 24 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 23 index: 50. Preceding OpId from leader: term: 24 index: 51. (index mismatch)
I20250114 03:11:48.502897  4534 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250114 03:11:48.503862  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 24 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 23 index: 50. Preceding OpId from leader: term: 24 index: 51. (index mismatch)
I20250114 03:11:48.504125  4538 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250114 03:11:48.605051  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:48.605146  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 24 FOLLOWER]: Advancing to term 25
I20250114 03:11:48.605798  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.605995  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 25 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:48.606196  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 25 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:48.606204  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 25 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:48.606429  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:48.606555  4533 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20250114 03:11:48.611989  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 25 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 24. Current term is 25. Ops: []
I20250114 03:11:48.612214  4532 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 24.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250114 03:11:48.612313  4532 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 24 LEADER]: Stepping down as leader of term 24
I20250114 03:11:48.612345  4532 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:48.612396  4532 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 24.51, Last appended by leader: 51, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.612495  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 24 FOLLOWER]: Advancing to term 25
I20250114 03:11:48.696825  4536 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:48.696933  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 24 FOLLOWER]: Advancing to term 25
I20250114 03:11:48.697641  4536 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.697803  4536 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 25 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:48.698035  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 25 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:48.698035  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 25 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:48.698170  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 25: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:48.698781  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 25.
I20250114 03:11:48.698982  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:48.699072  4536 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 25 FOLLOWER]: Leader election won for term 25
I20250114 03:11:48.699174  4536 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 25 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:48.699218  4536 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 24.51, Last appended by leader: 51, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.699616  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 24 to 25, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 25 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:48.713934  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:48.714054  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 25 FOLLOWER]: Advancing to term 26
I20250114 03:11:48.714747  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.714929  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 26 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:48.715138  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 26 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:48.715219  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 25 FOLLOWER]: Advancing to term 26
I20250114 03:11:48.715148  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 26 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:48.715811  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 26.
I20250114 03:11:48.716003  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:48.716092  4533 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 26 FOLLOWER]: Leader election won for term 26
I20250114 03:11:48.716161  4533 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 26 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:48.716221  4533 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 24.51, Last appended by leader: 51, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.716670  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 25 to 26, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 26 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:48.758090  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:48.776082  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 26 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 25. Current term is 26. Ops: []
I20250114 03:11:48.776391  4536 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250114 03:11:48.776525  4536 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 26.52 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I20250114 03:11:48.776583  4540 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 25 LEADER]: Stepping down as leader of term 25
I20250114 03:11:48.776638  4540 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:48.776705  4540 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 25.52, Last appended by leader: 52, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:48.776795  4540 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 25 FOLLOWER]: Advancing to term 26
I20250114 03:11:48.830015  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 26 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 25 index: 52. Preceding OpId from leader: term: 26 index: 52. (term mismatch)
I20250114 03:11:48.830096  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 51
I20250114 03:11:48.830118  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 25.52
I20250114 03:11:48.830140  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:48.830363  4533 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250114 03:11:48.831298  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 26 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 24 index: 51. Preceding OpId from leader: term: 26 index: 52. (index mismatch)
I20250114 03:11:48.831465  4539 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250114 03:11:49.103646  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:49.103755  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 26 FOLLOWER]: Advancing to term 27
I20250114 03:11:49.104414  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.104600  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 27 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:49.104833  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 27 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:49.104818  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 27 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:49.105141  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:49.105232  4532 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20250114 03:11:49.124039  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 27 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 26. Current term is 27. Ops: []
I20250114 03:11:49.124341  4541 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 26.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250114 03:11:49.124475  4533 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 26 LEADER]: Stepping down as leader of term 26
I20250114 03:11:49.124514  4533 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:49.124572  4533 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 26.52, Last appended by leader: 52, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.124665  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 26 FOLLOWER]: Advancing to term 27
I20250114 03:11:49.128407  4540 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:49.128486  4540 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 26 FOLLOWER]: Advancing to term 27
I20250114 03:11:49.129117  4540 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.129277  4540 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 27 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:49.129468  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 27 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:49.129468  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 27 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:49.129583  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 27: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:49.130278  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 27.
I20250114 03:11:49.130455  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:49.130563  4540 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 27 FOLLOWER]: Leader election won for term 27
I20250114 03:11:49.130621  4540 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 27 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:49.130677  4540 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 26.52, Last appended by leader: 52, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.131160  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 26 to 27, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 27 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:49.226656  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 27 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 26 index: 52. Preceding OpId from leader: term: 27 index: 53. (index mismatch)
I20250114 03:11:49.226963  4540 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250114 03:11:49.228120  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 27 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 26 index: 52. Preceding OpId from leader: term: 27 index: 53. (index mismatch)
I20250114 03:11:49.228320  4542 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250114 03:11:49.228489  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:49.228549  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 27 FOLLOWER]: Advancing to term 28
I20250114 03:11:49.229118  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.229220  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 28 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 27. Current term is 28. Ops: [27.53-27.53]
I20250114 03:11:49.229251  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 28 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:49.229377  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 28 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:49.229444  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 28 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:49.229442  4542 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 26.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250114 03:11:49.229614  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:49.229635  4542 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 27 LEADER]: Stepping down as leader of term 27
I20250114 03:11:49.229683  4542 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:49.229729  4532 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20250114 03:11:49.229748  4542 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 27.53, Last appended by leader: 53, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.229830  4542 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 27 FOLLOWER]: Advancing to term 28
I20250114 03:11:49.350369  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 27 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:49.350478  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 27 FOLLOWER]: Advancing to term 28
I20250114 03:11:49.351146  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.351320  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 28 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:49.351557  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 28 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:49.351578  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 28 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:49.351688  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 28: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:49.352196  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 28.
I20250114 03:11:49.352370  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:49.352469  4533 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 28 FOLLOWER]: Leader election won for term 28
I20250114 03:11:49.352531  4533 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 28 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:49.352587  4533 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 27.53, Last appended by leader: 53, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.353180  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 27 to 28, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 28 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:49.411230  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:49.411325  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 28 FOLLOWER]: Advancing to term 29
I20250114 03:11:49.411953  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.412115  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 29 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:49.412384  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 29 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:49.412395  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 29 candidate_status { last_received { term: 26 index: 52 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:49.412518  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 28 FOLLOWER]: Advancing to term 29
I20250114 03:11:49.413077  4273 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 for term 29 because replica has last-logged OpId of term: 27 index: 53, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 52.
I20250114 03:11:49.413285  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:49.413383  4532 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20250114 03:11:49.457428  4542 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:49.457437  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 29 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 28. Current term is 29. Ops: []
I20250114 03:11:49.457533  4542 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 29 FOLLOWER]: Advancing to term 30
I20250114 03:11:49.457768  4533 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 54, Last known committed idx: 52, Time since last communication: 0.000s
I20250114 03:11:49.457923  4541 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 28 LEADER]: Stepping down as leader of term 28
I20250114 03:11:49.457974  4541 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:49.458038  4541 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 28.54, Last appended by leader: 54, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.458142  4541 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 28 FOLLOWER]: Advancing to term 29
I20250114 03:11:49.458443  4542 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.458564  4542 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 30 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:49.458746  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 30 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:49.458752  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 30 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:49.458850  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 29 FOLLOWER]: Advancing to term 30
I20250114 03:11:49.458887  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 29 FOLLOWER]: Advancing to term 30
I20250114 03:11:49.459367  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 30.
I20250114 03:11:49.459477  4425 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 for term 30 because replica has last-logged OpId of term: 28 index: 54, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 53.
I20250114 03:11:49.459535  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 
I20250114 03:11:49.459620  4542 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 30 FOLLOWER]: Leader election won for term 30
I20250114 03:11:49.459674  4542 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 30 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:49.459731  4542 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 27.53, Last appended by leader: 53, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.460188  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 28 to 30, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 30 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:49.549724  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 30 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 26 index: 52. Preceding OpId from leader: term: 30 index: 54. (index mismatch)
I20250114 03:11:49.550060  4542 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 52, Time since last communication: 0.000s
I20250114 03:11:49.551183  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 30 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 28 index: 54. Preceding OpId from leader: term: 30 index: 54. (term mismatch)
I20250114 03:11:49.551240  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 53
I20250114 03:11:49.551262  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 28.54
I20250114 03:11:49.551296  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:49.551474  4542 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250114 03:11:49.651903  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:49.652014  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 30 FOLLOWER]: Advancing to term 31
I20250114 03:11:49.652667  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.652832  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 31 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:49.653050  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 31 candidate_status { last_received { term: 30 index: 54 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:49.653050  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 31 candidate_status { last_received { term: 30 index: 54 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:49.653157  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 30 FOLLOWER]: Advancing to term 31
I20250114 03:11:49.653724  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 31.
I20250114 03:11:49.653911  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:49.653992  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 31 FOLLOWER]: Leader election won for term 31
I20250114 03:11:49.654102  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 31 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:49.654148  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 30.54, Last appended by leader: 54, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.654561  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 30 to 31, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 31 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:49.662823  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 31 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 30. Current term is 31. Ops: []
I20250114 03:11:49.663085  4536 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 30.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:11:49.663192  4536 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 30 LEADER]: Stepping down as leader of term 30
I20250114 03:11:49.663225  4536 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:49.663280  4536 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 30.54, Last appended by leader: 54, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.663349  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 30 FOLLOWER]: Advancing to term 31
I20250114 03:11:49.739114  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 31 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 30 index: 54. Preceding OpId from leader: term: 31 index: 55. (index mismatch)
I20250114 03:11:49.739395  4532 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:11:49.740693  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 31 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 30 index: 54. Preceding OpId from leader: term: 31 index: 55. (index mismatch)
I20250114 03:11:49.740983  4544 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:11:49.758358  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:49.840946  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:49.841055  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 31 FOLLOWER]: Advancing to term 32
I20250114 03:11:49.841768  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.841941  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 32 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:49.842098  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:49.842145  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:49.842093  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 32 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 31. Current term is 32. Ops: []
I20250114 03:11:49.842391  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:49.842399  4543 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 31.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250114 03:11:49.842489  4533 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250114 03:11:49.842548  4543 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 31 LEADER]: Stepping down as leader of term 31
I20250114 03:11:49.842589  4543 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:49.842653  4543 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 31.55, Last appended by leader: 55, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.842731  4543 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 31 FOLLOWER]: Advancing to term 32
I20250114 03:11:49.925171  4542 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:49.925282  4542 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 31 FOLLOWER]: Advancing to term 32
I20250114 03:11:49.926015  4542 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.926182  4542 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 32 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:49.926411  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:49.926450  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:49.926569  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 32: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:49.927052  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 32.
I20250114 03:11:49.927234  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:49.927320  4542 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 32 FOLLOWER]: Leader election won for term 32
I20250114 03:11:49.927373  4542 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 32 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:49.927412  4542 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 31.55, Last appended by leader: 55, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.927922  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 31 to 32, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 32 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:49.946887  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:49.946993  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 32 FOLLOWER]: Advancing to term 33
I20250114 03:11:49.947849  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.948042  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 33 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:49.948240  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 33 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:49.948240  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 33 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:49.948335  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 32 FOLLOWER]: Advancing to term 33
I20250114 03:11:49.948881  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 33.
I20250114 03:11:49.949069  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:49.949175  4533 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 33 FOLLOWER]: Leader election won for term 33
I20250114 03:11:49.949234  4533 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 33 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:49.949301  4533 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 31.55, Last appended by leader: 55, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:49.949748  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 32 to 33, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 33 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:50.003382  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 33 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 32. Current term is 33. Ops: []
I20250114 03:11:50.003731  4542 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250114 03:11:50.003873  4542 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 32 LEADER]: Stepping down as leader of term 32
I20250114 03:11:50.003916  4542 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:50.003975  4542 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 32.56, Last appended by leader: 56, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.004055  4542 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 32 FOLLOWER]: Advancing to term 33
I20250114 03:11:50.038192  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 33 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 31 index: 55. Preceding OpId from leader: term: 33 index: 56. (index mismatch)
I20250114 03:11:50.038497  4533 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250114 03:11:50.039631  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 33 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 32 index: 56. Preceding OpId from leader: term: 33 index: 56. (term mismatch)
I20250114 03:11:50.039686  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 55
I20250114 03:11:50.039724  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 32.56
I20250114 03:11:50.039747  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:50.040016  4545 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250114 03:11:50.140337  4543 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:50.140436  4543 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 33 FOLLOWER]: Advancing to term 34
I20250114 03:11:50.141115  4543 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.141317  4543 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 34 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:50.141542  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 34 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:50.141626  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 33 FOLLOWER]: Advancing to term 34
I20250114 03:11:50.141548  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 34 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:50.142298  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 34.
I20250114 03:11:50.142448  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 34 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 33. Current term is 34. Ops: []
I20250114 03:11:50.142519  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:50.142621  4543 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 34 FOLLOWER]: Leader election won for term 34
I20250114 03:11:50.142678  4543 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 34 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:50.142729  4543 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 33.56, Last appended by leader: 56, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.142796  4545 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 33.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250114 03:11:50.142915  4545 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 33 LEADER]: Stepping down as leader of term 33
I20250114 03:11:50.142973  4545 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:50.143041  4545 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 33.56, Last appended by leader: 56, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.143133  4545 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 33 FOLLOWER]: Advancing to term 34
I20250114 03:11:50.143270  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 33 to 34, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 34 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:50.218719  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 34 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 33 index: 56. Preceding OpId from leader: term: 34 index: 57. (index mismatch)
I20250114 03:11:50.219132  4543 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250114 03:11:50.220425  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 34 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 33 index: 56. Preceding OpId from leader: term: 34 index: 57. (index mismatch)
I20250114 03:11:50.220690  4544 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250114 03:11:50.414556  4545 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 34 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:50.414666  4545 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 34 FOLLOWER]: Advancing to term 35
I20250114 03:11:50.415336  4545 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.415508  4545 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 35 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:50.415700  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 35 candidate_status { last_received { term: 34 index: 57 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:50.415704  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 35 candidate_status { last_received { term: 34 index: 57 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:50.415936  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:50.416018  4545 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20250114 03:11:50.420135  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 35 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 34. Current term is 35. Ops: []
I20250114 03:11:50.420363  4544 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 34.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250114 03:11:50.420485  4532 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 34 LEADER]: Stepping down as leader of term 34
I20250114 03:11:50.420542  4532 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:50.420619  4532 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 34.57, Last appended by leader: 57, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.420696  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 34 FOLLOWER]: Advancing to term 35
I20250114 03:11:50.493258  4542 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:50.493369  4542 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 34 FOLLOWER]: Advancing to term 35
I20250114 03:11:50.494110  4542 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.494274  4542 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 35 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:50.494549  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 35 candidate_status { last_received { term: 34 index: 57 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:50.494549  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 35 candidate_status { last_received { term: 34 index: 57 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:50.494690  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 35: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:50.495244  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 35.
I20250114 03:11:50.495437  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:50.495576  4542 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 35 FOLLOWER]: Leader election won for term 35
I20250114 03:11:50.495654  4542 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 35 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:50.495707  4542 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 34.57, Last appended by leader: 57, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.496282  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 34 to 35, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 35 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:50.530093  4545 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:50.530205  4545 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 35 FOLLOWER]: Advancing to term 36
I20250114 03:11:50.530900  4545 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.531080  4545 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 36 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:50.531307  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 36 candidate_status { last_received { term: 34 index: 57 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:50.531380  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 35 FOLLOWER]: Advancing to term 36
I20250114 03:11:50.531312  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 36 candidate_status { last_received { term: 34 index: 57 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:50.531965  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 36.
I20250114 03:11:50.532162  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:50.532248  4545 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 36 FOLLOWER]: Leader election won for term 36
I20250114 03:11:50.532298  4545 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 36 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:50.532354  4545 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 34.57, Last appended by leader: 57, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.532841  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 35 to 36, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 36 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:50.596968  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 36 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 35. Current term is 36. Ops: []
I20250114 03:11:50.597254  4542 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250114 03:11:50.597349  4542 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 36.58 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 57
I20250114 03:11:50.597381  4536 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 35 LEADER]: Stepping down as leader of term 35
I20250114 03:11:50.597420  4536 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:50.597471  4536 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 35.58, Last appended by leader: 58, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.597566  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 35 FOLLOWER]: Advancing to term 36
I20250114 03:11:50.623394  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 36 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 35 index: 58. Preceding OpId from leader: term: 36 index: 58. (term mismatch)
I20250114 03:11:50.623471  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 57
I20250114 03:11:50.623490  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 35.58
I20250114 03:11:50.623510  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:50.623793  4545 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250114 03:11:50.624954  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 36 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 34 index: 57. Preceding OpId from leader: term: 36 index: 58. (index mismatch)
I20250114 03:11:50.625142  4545 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250114 03:11:50.758610  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:50.814321  4544 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:50.814401  4544 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 36 FOLLOWER]: Advancing to term 37
I20250114 03:11:50.815110  4544 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.815268  4544 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 37 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:50.815485  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 37 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:50.815492  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 37 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:50.815776  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:50.815861  4544 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20250114 03:11:50.825911  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 37 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 36. Current term is 37. Ops: []
I20250114 03:11:50.826161  4546 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 36.58, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250114 03:11:50.826283  4533 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 36 LEADER]: Stepping down as leader of term 36
I20250114 03:11:50.826324  4533 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:50.826380  4533 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 36.58, Last appended by leader: 58, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.826465  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 36 FOLLOWER]: Advancing to term 37
I20250114 03:11:50.907030  4536 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:50.907133  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 36 FOLLOWER]: Advancing to term 37
I20250114 03:11:50.907822  4536 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.907997  4536 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 37 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:50.908226  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 37 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:50.908238  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 37 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:50.908360  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 37: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:50.908908  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 37.
I20250114 03:11:50.909067  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:50.909150  4536 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 37 FOLLOWER]: Leader election won for term 37
I20250114 03:11:50.909197  4536 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 37 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:50.909251  4536 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 36.58, Last appended by leader: 58, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.909737  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 36 to 37, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 37 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:50.947499  4544 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:50.947628  4544 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 37 FOLLOWER]: Advancing to term 38
I20250114 03:11:50.948323  4544 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.948501  4544 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 38 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:50.948714  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 38 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:50.948745  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 38 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:50.948796  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 37 FOLLOWER]: Advancing to term 38
I20250114 03:11:50.949393  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 38.
I20250114 03:11:50.949596  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:50.949682  4544 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 38 FOLLOWER]: Leader election won for term 38
I20250114 03:11:50.949733  4544 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 38 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:50.949787  4544 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 36.58, Last appended by leader: 58, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.950263  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 37 to 38, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 38 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:50.989784  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 38 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 37. Current term is 38. Ops: []
I20250114 03:11:50.990010  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 38 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 37. Current term is 38. Ops: []
I20250114 03:11:50.990123  4536 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250114 03:11:50.990283  4536 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 38.59 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 58
I20250114 03:11:50.990326  4547 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 37 LEADER]: Stepping down as leader of term 37
I20250114 03:11:50.990363  4547 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:50.990347  4542 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250114 03:11:50.990495  4547 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 37.59, Last appended by leader: 59, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:50.990583  4547 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 37 FOLLOWER]: Advancing to term 38
I20250114 03:11:51.034802  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 38 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 37 index: 59. Preceding OpId from leader: term: 38 index: 59. (term mismatch)
I20250114 03:11:51.034888  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 58
I20250114 03:11:51.034921  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 37.59
I20250114 03:11:51.034943  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:51.035233  4544 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250114 03:11:51.036446  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 38 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 36 index: 58. Preceding OpId from leader: term: 38 index: 59. (index mismatch)
I20250114 03:11:51.036638  4544 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250114 03:11:51.490703  4536 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:51.490834  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 38 FOLLOWER]: Advancing to term 39
I20250114 03:11:51.491503  4536 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.491672  4536 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 39 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:51.491880  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 39 candidate_status { last_received { term: 38 index: 59 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:51.491894  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 39 candidate_status { last_received { term: 38 index: 59 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:51.492110  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:51.492237  4536 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20250114 03:11:51.497295  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 39 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 38. Current term is 39. Ops: []
I20250114 03:11:51.497515  4548 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 38.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250114 03:11:51.497644  4532 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 38 LEADER]: Stepping down as leader of term 38
I20250114 03:11:51.497689  4532 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:51.497746  4532 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 38.59, Last appended by leader: 59, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.497830  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 38 FOLLOWER]: Advancing to term 39
I20250114 03:11:51.501037  4546 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 38 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:51.501096  4546 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 38 FOLLOWER]: Advancing to term 39
I20250114 03:11:51.501727  4546 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.501883  4546 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 39 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:51.502056  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 39 candidate_status { last_received { term: 38 index: 59 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:51.502116  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 39: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:51.502080  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 39 candidate_status { last_received { term: 38 index: 59 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:51.502753  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 39.
I20250114 03:11:51.502943  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:51.503039  4546 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 39 FOLLOWER]: Leader election won for term 39
I20250114 03:11:51.503098  4546 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 39 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:51.503150  4546 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 38.59, Last appended by leader: 59, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.503587  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 38 to 39, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 39 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:51.584087  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 39 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 38 index: 59. Preceding OpId from leader: term: 39 index: 60. (index mismatch)
I20250114 03:11:51.584390  4546 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250114 03:11:51.585528  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 39 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 38 index: 59. Preceding OpId from leader: term: 39 index: 60. (index mismatch)
I20250114 03:11:51.585726  4546 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250114 03:11:51.758890  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:51.803464  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:51.803579  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 39 FOLLOWER]: Advancing to term 40
I20250114 03:11:51.804201  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.804360  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 40 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:51.804634  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 40 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:51.804641  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 40 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:51.804917  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:51.805004  4532 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20250114 03:11:51.807699  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 40 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 39. Current term is 40. Ops: []
I20250114 03:11:51.807981  4546 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 39.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250114 03:11:51.808108  4546 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 39 LEADER]: Stepping down as leader of term 39
I20250114 03:11:51.808151  4546 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:51.808197  4546 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 39.60, Last appended by leader: 60, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.808280  4546 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 39 FOLLOWER]: Advancing to term 40
I20250114 03:11:51.873400  4536 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:51.873497  4536 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 39 FOLLOWER]: Advancing to term 40
I20250114 03:11:51.874207  4536 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.874372  4536 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 40 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:51.874574  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 40 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:51.874579  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 40 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:51.874709  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 40: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:51.875232  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 40.
I20250114 03:11:51.875418  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:51.875519  4536 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 40 FOLLOWER]: Leader election won for term 40
I20250114 03:11:51.875584  4536 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 40 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:51.875640  4536 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 39.60, Last appended by leader: 60, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.876078  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 39 to 40, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 40 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:51.934585  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:51.934692  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 40 FOLLOWER]: Advancing to term 41
I20250114 03:11:51.935410  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.935565  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 41 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:51.935811  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 41 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:51.935823  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 41 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:51.935916  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 40 FOLLOWER]: Advancing to term 41
I20250114 03:11:51.936439  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 41.
I20250114 03:11:51.936650  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:51.936746  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 41 FOLLOWER]: Leader election won for term 41
I20250114 03:11:51.936820  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 41 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:51.936872  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 39.60, Last appended by leader: 60, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.937290  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 40 to 41, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 41 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:51.968533  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 41 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 40. Current term is 41. Ops: []
I20250114 03:11:51.968834  4536 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250114 03:11:51.968947  4536 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 41.61 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 60
I20250114 03:11:51.968989  4547 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 40 LEADER]: Stepping down as leader of term 40
I20250114 03:11:51.969030  4547 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:51.969084  4547 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 40.61, Last appended by leader: 61, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:51.969166  4547 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 40 FOLLOWER]: Advancing to term 41
I20250114 03:11:52.019621  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 41 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 39 index: 60. Preceding OpId from leader: term: 41 index: 61. (index mismatch)
I20250114 03:11:52.019922  4532 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250114 03:11:52.020838  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 41 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 40 index: 61. Preceding OpId from leader: term: 41 index: 61. (term mismatch)
I20250114 03:11:52.020890  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 60
I20250114 03:11:52.020910  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 40.61
I20250114 03:11:52.020927  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:52.021092  4548 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250114 03:11:52.121646  4546 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 41 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:52.121747  4546 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 41 FOLLOWER]: Advancing to term 42
I20250114 03:11:52.122439  4546 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.122596  4546 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 42 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:52.122839  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 42 candidate_status { last_received { term: 41 index: 61 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:52.122851  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 42 candidate_status { last_received { term: 41 index: 61 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:52.122937  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 41 FOLLOWER]: Advancing to term 42
I20250114 03:11:52.123456  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 42.
I20250114 03:11:52.123667  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:52.123768  4546 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 42 FOLLOWER]: Leader election won for term 42
I20250114 03:11:52.123836  4546 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 42 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:52.123889  4546 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 41.61, Last appended by leader: 61, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.124434  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 41 to 42, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 42 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:52.125217  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 42 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 41. Current term is 42. Ops: []
I20250114 03:11:52.125391  4548 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 41.61, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250114 03:11:52.125463  4548 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 42.62 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 61
I20250114 03:11:52.125484  4548 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 41 LEADER]: Stepping down as leader of term 41
I20250114 03:11:52.125511  4548 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:52.125566  4548 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 41.61, Last appended by leader: 61, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.125641  4548 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 41 FOLLOWER]: Advancing to term 42
I20250114 03:11:52.221803  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 42 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 41 index: 61. Preceding OpId from leader: term: 42 index: 62. (index mismatch)
I20250114 03:11:52.222079  4546 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250114 03:11:52.223078  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 42 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 41 index: 61. Preceding OpId from leader: term: 42 index: 62. (index mismatch)
I20250114 03:11:52.223297  4552 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250114 03:11:52.510077  4547 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:52.510174  4547 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 42 FOLLOWER]: Advancing to term 43
I20250114 03:11:52.510859  4547 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.511036  4547 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 43 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:52.511330  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 43 candidate_status { last_received { term: 42 index: 62 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:52.511332  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 43 candidate_status { last_received { term: 42 index: 62 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:52.511600  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 43 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 42. Current term is 43. Ops: []
I20250114 03:11:52.511631  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:52.511715  4547 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20250114 03:11:52.511878  4533 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 42.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250114 03:11:52.511986  4552 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 42 LEADER]: Stepping down as leader of term 42
I20250114 03:11:52.512024  4552 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:52.512085  4552 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 42.62, Last appended by leader: 62, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.512161  4552 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 42 FOLLOWER]: Advancing to term 43
I20250114 03:11:52.571041  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:52.571151  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 42 FOLLOWER]: Advancing to term 43
I20250114 03:11:52.571861  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.572042  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 43 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:52.572291  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 43 candidate_status { last_received { term: 42 index: 62 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:52.572291  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 43 candidate_status { last_received { term: 42 index: 62 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:52.572407  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 43: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:52.572928  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 43.
I20250114 03:11:52.573134  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:52.573243  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 43 FOLLOWER]: Leader election won for term 43
I20250114 03:11:52.573297  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 43 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:52.573336  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 42.62, Last appended by leader: 62, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.573913  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 42 to 43, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 43 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:52.650210  4547 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:52.650321  4547 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 43 FOLLOWER]: Advancing to term 44
I20250114 03:11:52.651042  4547 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.651227  4547 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 44 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:52.651440  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 44 candidate_status { last_received { term: 42 index: 62 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:52.651520  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 43 FOLLOWER]: Advancing to term 44
I20250114 03:11:52.651454  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 44 candidate_status { last_received { term: 42 index: 62 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:52.652107  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 44.
I20250114 03:11:52.652271  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:52.652364  4547 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 44 FOLLOWER]: Leader election won for term 44
I20250114 03:11:52.652468  4547 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 44 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:52.652513  4547 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 42.62, Last appended by leader: 62, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.652945  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 43 to 44, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 44 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:52.668407  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 44 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 43. Current term is 44. Ops: []
I20250114 03:11:52.668686  4532 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250114 03:11:52.668810  4532 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 44.63 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 62
I20250114 03:11:52.668859  4548 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 43 LEADER]: Stepping down as leader of term 43
I20250114 03:11:52.668912  4548 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:52.668984  4548 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 43.63, Last appended by leader: 63, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.669059  4548 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 43 FOLLOWER]: Advancing to term 44
I20250114 03:11:52.732591  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 44 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 43 index: 63. Preceding OpId from leader: term: 44 index: 63. (term mismatch)
I20250114 03:11:52.732673  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 62
I20250114 03:11:52.732697  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 43.63
I20250114 03:11:52.732718  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:52.732976  4547 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250114 03:11:52.733956  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 44 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 42 index: 62. Preceding OpId from leader: term: 44 index: 63. (index mismatch)
I20250114 03:11:52.734198  4554 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250114 03:11:52.759218  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:52.835146  4552 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 44 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:52.835247  4552 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 44 FOLLOWER]: Advancing to term 45
I20250114 03:11:52.835945  4552 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.836128  4552 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 45 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:52.836367  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 45 candidate_status { last_received { term: 44 index: 63 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:52.836370  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 45 candidate_status { last_received { term: 44 index: 63 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:52.836462  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 44 FOLLOWER]: Advancing to term 45
I20250114 03:11:52.836997  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 45.
I20250114 03:11:52.837162  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:52.837249  4552 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 45 FOLLOWER]: Leader election won for term 45
I20250114 03:11:52.837312  4552 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 45 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:52.837375  4552 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 44.63, Last appended by leader: 63, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.837846  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 44 to 45, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 45 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:52.842578  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 45 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 44. Current term is 45. Ops: []
I20250114 03:11:52.842824  4554 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 44.63, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250114 03:11:52.842906  4554 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 44 LEADER]: Stepping down as leader of term 44
I20250114 03:11:52.842939  4554 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:52.842999  4554 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 44.63, Last appended by leader: 63, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.843076  4554 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 44 FOLLOWER]: Advancing to term 45
I20250114 03:11:52.940043  4548 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:52.940143  4548 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 45 FOLLOWER]: Advancing to term 46
I20250114 03:11:52.940851  4548 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.941030  4548 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 46 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:52.941241  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 46 candidate_status { last_received { term: 44 index: 63 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:52.941255  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 46 candidate_status { last_received { term: 44 index: 63 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:52.941332  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 45 FOLLOWER]: Advancing to term 46
I20250114 03:11:52.941973  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 46.
I20250114 03:11:52.942147  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:52.942235  4548 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 46 FOLLOWER]: Leader election won for term 46
I20250114 03:11:52.942286  4548 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 46 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:52.942342  4548 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 44.63, Last appended by leader: 63, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.942788  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 45 to 46, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 46 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:52.948186  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 46 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 45. Current term is 46. Ops: []
I20250114 03:11:52.948429  4552 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250114 03:11:52.948544  4552 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 45 LEADER]: Stepping down as leader of term 45
I20250114 03:11:52.948580  4552 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:52.948701  4552 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 45.64, Last appended by leader: 64, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:52.948796  4552 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 45 FOLLOWER]: Advancing to term 46
I20250114 03:11:53.048418  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 46 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 44 index: 63. Preceding OpId from leader: term: 46 index: 64. (index mismatch)
I20250114 03:11:53.048750  4548 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250114 03:11:53.049767  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 46 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 45 index: 64. Preceding OpId from leader: term: 46 index: 64. (term mismatch)
I20250114 03:11:53.049822  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 63
I20250114 03:11:53.049844  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 45.64
I20250114 03:11:53.049883  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:53.050046  4548 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250114 03:11:53.150626  4554 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:53.150720  4554 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 46 FOLLOWER]: Advancing to term 47
I20250114 03:11:53.151386  4554 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.151558  4554 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 47 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:53.151767  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 47 candidate_status { last_received { term: 46 index: 64 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:53.151775  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 47 candidate_status { last_received { term: 46 index: 64 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:53.151839  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 46 FOLLOWER]: Advancing to term 47
I20250114 03:11:53.152415  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 47.
I20250114 03:11:53.152585  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:53.152679  4554 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 47 FOLLOWER]: Leader election won for term 47
I20250114 03:11:53.152742  4554 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 47 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:53.152796  4554 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 46.64, Last appended by leader: 64, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.153236  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 46 to 47, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 47 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:53.163496  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 47 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 46. Current term is 47. Ops: []
I20250114 03:11:53.163781  4548 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 46.64, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250114 03:11:53.163879  4548 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 47.65 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 64
I20250114 03:11:53.163921  4532 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 46 LEADER]: Stepping down as leader of term 46
I20250114 03:11:53.163960  4532 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:53.164014  4532 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 46.64, Last appended by leader: 64, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.164088  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 46 FOLLOWER]: Advancing to term 47
I20250114 03:11:53.248485  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 47 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 46 index: 64. Preceding OpId from leader: term: 47 index: 65. (index mismatch)
I20250114 03:11:53.248757  4554 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250114 03:11:53.249815  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 47 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 46 index: 64. Preceding OpId from leader: term: 47 index: 65. (index mismatch)
I20250114 03:11:53.250154  4554 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250114 03:11:53.350666  4533 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 47 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:53.350781  4533 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 47 FOLLOWER]: Advancing to term 48
I20250114 03:11:53.351472  4533 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.351650  4533 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 48 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:53.351898  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 48 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:53.351899  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 48 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:53.352001  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 47 FOLLOWER]: Advancing to term 48
I20250114 03:11:53.352578  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 48.
I20250114 03:11:53.352809  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:53.352932  4533 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 48 FOLLOWER]: Leader election won for term 48
I20250114 03:11:53.353003  4533 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 48 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:53.353061  4533 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 47.65, Last appended by leader: 65, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.353488  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 47 to 48, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 48 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:53.354432  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 48 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 47. Current term is 48. Ops: []
I20250114 03:11:53.354658  4554 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 47.65, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250114 03:11:53.354759  4554 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 47 LEADER]: Stepping down as leader of term 47
I20250114 03:11:53.354794  4554 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:53.354854  4554 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 47.65, Last appended by leader: 65, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.354945  4554 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 47 FOLLOWER]: Advancing to term 48
I20250114 03:11:53.439865  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 48 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 47 index: 65. Preceding OpId from leader: term: 48 index: 66. (index mismatch)
I20250114 03:11:53.440131  4533 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250114 03:11:53.441203  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 48 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 47 index: 65. Preceding OpId from leader: term: 48 index: 66. (index mismatch)
I20250114 03:11:53.441424  4555 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250114 03:11:53.719862  4554 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:53.719960  4554 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 48 FOLLOWER]: Advancing to term 49
I20250114 03:11:53.720659  4554 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.720917  4554 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 49 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:53.721050  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 49 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:53.721055  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 49 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:53.721325  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:53.721419  4554 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20250114 03:11:53.735864  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 49 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 48. Current term is 49. Ops: []
I20250114 03:11:53.736142  4555 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 48.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250114 03:11:53.736289  4555 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 48 LEADER]: Stepping down as leader of term 48
I20250114 03:11:53.736333  4555 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:53.736392  4555 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 48.66, Last appended by leader: 66, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.736485  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 48 FOLLOWER]: Advancing to term 49
I20250114 03:11:53.759513  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:53.796622  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:53.796729  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 48 FOLLOWER]: Advancing to term 49
I20250114 03:11:53.797358  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.797541  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 49 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:53.797767  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 49 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:53.797780  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 49 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:53.797897  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 49: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:53.798460  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 49.
I20250114 03:11:53.798688  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:53.798802  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 49 FOLLOWER]: Leader election won for term 49
I20250114 03:11:53.798935  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 49 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:53.798983  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 48.66, Last appended by leader: 66, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.799469  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 48 to 49, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 49 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:53.837936  4554 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:53.838039  4554 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 49 FOLLOWER]: Advancing to term 50
I20250114 03:11:53.838758  4554 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.838961  4554 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 50 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:53.839146  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 50 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:53.839218  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 49 FOLLOWER]: Advancing to term 50
I20250114 03:11:53.839152  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 50 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:53.839763  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 50.
I20250114 03:11:53.839942  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:53.840029  4554 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 50 FOLLOWER]: Leader election won for term 50
I20250114 03:11:53.840093  4554 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 50 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:53.840150  4554 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 48.66, Last appended by leader: 66, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.840601  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 49 to 50, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 50 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:53.907130  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 50 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 49. Current term is 50. Ops: []
I20250114 03:11:53.907454  4532 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250114 03:11:53.907567  4532 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 50.67 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 66
I20250114 03:11:53.907610  4532 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 49 LEADER]: Stepping down as leader of term 49
I20250114 03:11:53.907625  4532 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:53.907707  4532 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 49.67, Last appended by leader: 67, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:53.907807  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 49 FOLLOWER]: Advancing to term 50
I20250114 03:11:53.917078  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 50 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 49 index: 67. Preceding OpId from leader: term: 50 index: 67. (term mismatch)
I20250114 03:11:53.917158  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 66
I20250114 03:11:53.917181  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 49.67
I20250114 03:11:53.917203  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:53.917479  4554 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250114 03:11:53.918843  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 50 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 48 index: 66. Preceding OpId from leader: term: 50 index: 67. (index mismatch)
I20250114 03:11:53.919032  4554 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250114 03:11:54.019958  4552 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 50 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:54.020059  4552 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 50 FOLLOWER]: Advancing to term 51
I20250114 03:11:54.020711  4552 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.020910  4552 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 51 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:54.021098  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 51 candidate_status { last_received { term: 50 index: 67 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:54.021113  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 51 candidate_status { last_received { term: 50 index: 67 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:54.021330  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:11:54.021493  4552 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20250114 03:11:54.041357  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 51 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 50. Current term is 51. Ops: []
I20250114 03:11:54.041643  4557 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 50.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250114 03:11:54.041757  4557 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 50 LEADER]: Stepping down as leader of term 50
I20250114 03:11:54.041785  4557 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:54.041857  4557 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 50.67, Last appended by leader: 67, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.041965  4557 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 50 FOLLOWER]: Advancing to term 51
I20250114 03:11:54.133716  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:54.133833  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 50 FOLLOWER]: Advancing to term 51
I20250114 03:11:54.134501  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.134683  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 51 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:54.134896  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 51 candidate_status { last_received { term: 50 index: 67 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:54.134896  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 51 candidate_status { last_received { term: 50 index: 67 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:54.135041  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 51: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:11:54.135607  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 51.
I20250114 03:11:54.135756  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:54.135843  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 51 FOLLOWER]: Leader election won for term 51
I20250114 03:11:54.135911  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 51 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:54.135967  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 50.67, Last appended by leader: 67, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.136415  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 50 to 51, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 51 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:54.136749  4552 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:54.136804  4552 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 51 FOLLOWER]: Advancing to term 52
I20250114 03:11:54.137382  4552 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.137501  4552 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 52 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:54.137615  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 52 candidate_status { last_received { term: 50 index: 67 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:54.137650  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 52 candidate_status { last_received { term: 50 index: 67 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:54.137676  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 51 FOLLOWER]: Advancing to term 52
I20250114 03:11:54.138178  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 52.
I20250114 03:11:54.138311  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:54.138406  4552 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 52 FOLLOWER]: Leader election won for term 52
I20250114 03:11:54.138460  4552 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 52 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:54.138522  4552 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 50.67, Last appended by leader: 67, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.138867  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 51 to 52, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 52 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:54.216797  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 52 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 51. Current term is 52. Ops: []
I20250114 03:11:54.216969  4347 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 51 LEADER]: Stepping down as leader of term 51
I20250114 03:11:54.217026  4347 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:54.217087  4347 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 51.68, Last appended by leader: 68, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.217171  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 51 FOLLOWER]: Advancing to term 52
W20250114 03:11:54.217190  4532 consensus_queue.cc:1159] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5eea63c188224be98e6888084ea42808" responder_term: 52 status { last_received { term: 50 index: 67 } last_committed_idx: 67 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 51. Current term is 52. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250114 03:11:54.217837  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 52 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 51 index: 68. Preceding OpId from leader: term: 52 index: 68. (term mismatch)
I20250114 03:11:54.217914  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 67
I20250114 03:11:54.217938  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 51.68
I20250114 03:11:54.217973  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 52 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:54.218197  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 52 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 50 index: 67. Preceding OpId from leader: term: 52 index: 68. (index mismatch)
I20250114 03:11:54.218216  4552 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250114 03:11:54.218412  4552 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250114 03:11:54.409004  4557 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:54.409102  4557 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 52 FOLLOWER]: Advancing to term 53
I20250114 03:11:54.409835  4557 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.410041  4557 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 53 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:54.410502  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 53 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:54.410502  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 53 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:54.410776  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:54.410857  4557 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20250114 03:11:54.420950  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 53 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 52. Current term is 53. Ops: []
I20250114 03:11:54.421186  4555 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 52.68, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250114 03:11:54.421301  4555 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 52 LEADER]: Stepping down as leader of term 52
I20250114 03:11:54.421341  4555 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:54.421396  4555 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 52.68, Last appended by leader: 68, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.421471  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 52 FOLLOWER]: Advancing to term 53
I20250114 03:11:54.442972  4532 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:54.443061  4532 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 52 FOLLOWER]: Advancing to term 53
I20250114 03:11:54.443655  4532 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.443804  4532 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 53 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:54.444026  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 53 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:54.444026  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 53 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:54.444129  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 53: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:54.444696  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 53.
I20250114 03:11:54.444866  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:54.444942  4532 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 53 FOLLOWER]: Leader election won for term 53
I20250114 03:11:54.444994  4532 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 53 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:54.445051  4532 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 52.68, Last appended by leader: 68, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.445497  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 52 to 53, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 53 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:54.526994  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 53 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 52 index: 68. Preceding OpId from leader: term: 53 index: 69. (index mismatch)
I20250114 03:11:54.527267  4532 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250114 03:11:54.528259  4557 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:54.528317  4557 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 53 FOLLOWER]: Advancing to term 54
I20250114 03:11:54.529069  4557 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.529178  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 54 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 53. Current term is 54. Ops: []
I20250114 03:11:54.529212  4557 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 54 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:54.529321  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 54 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:54.529630  4556 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250114 03:11:54.529667  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 54 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:54.529719  4556 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 53 LEADER]: Stepping down as leader of term 53
I20250114 03:11:54.529788  4556 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:54.529939  4556 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 53.69, Last appended by leader: 69, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.530025  4556 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 53 FOLLOWER]: Advancing to term 54
I20250114 03:11:54.530673  4347 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 for term 54 because replica has last-logged OpId of term: 53 index: 69, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 68.
I20250114 03:11:54.530877  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:54.530972  4557 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20250114 03:11:54.647101  4555 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 53 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:54.647205  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 53 FOLLOWER]: Advancing to term 54
I20250114 03:11:54.647886  4555 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.648048  4555 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 54 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:54.648293  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 54 candidate_status { last_received { term: 53 index: 69 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:54.648293  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 54 candidate_status { last_received { term: 53 index: 69 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:54.648444  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 54: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:54.649021  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 54.
I20250114 03:11:54.649199  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:54.649294  4555 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 54 FOLLOWER]: Leader election won for term 54
I20250114 03:11:54.649355  4555 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 54 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:54.649394  4555 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 53.69, Last appended by leader: 69, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.649852  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 53 to 54, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 54 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:54.731477  4557 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:54.731578  4557 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 54 FOLLOWER]: Advancing to term 55
I20250114 03:11:54.732270  4557 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.732434  4557 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 55 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:54.732663  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 55 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:54.732746  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 54 FOLLOWER]: Advancing to term 55
I20250114 03:11:54.732666  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 55 candidate_status { last_received { term: 52 index: 68 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:54.733336  4347 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 for term 55 because replica has last-logged OpId of term: 53 index: 69, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 68.
I20250114 03:11:54.733533  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:54.733619  4557 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20250114 03:11:54.747282  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 55 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 54. Current term is 55. Ops: []
I20250114 03:11:54.747575  4555 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250114 03:11:54.747689  4555 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 54 LEADER]: Stepping down as leader of term 54
I20250114 03:11:54.747722  4555 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:54.747812  4555 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 54.70, Last appended by leader: 70, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.747889  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 54 FOLLOWER]: Advancing to term 55
I20250114 03:11:54.759779  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:54.791896  4556 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:54.792006  4556 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 55 FOLLOWER]: Advancing to term 56
I20250114 03:11:54.792764  4556 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.792938  4556 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 56 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:54.793147  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 56 candidate_status { last_received { term: 53 index: 69 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:54.793219  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 55 FOLLOWER]: Advancing to term 56
I20250114 03:11:54.793151  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 56 candidate_status { last_received { term: 53 index: 69 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:54.793282  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 55 FOLLOWER]: Advancing to term 56
I20250114 03:11:54.793815  4425 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 for term 56 because replica has last-logged OpId of term: 54 index: 70, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 69.
I20250114 03:11:54.793844  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 56.
I20250114 03:11:54.794042  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:54.794152  4556 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 56 FOLLOWER]: Leader election won for term 56
I20250114 03:11:54.794215  4556 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 56 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:54.794276  4556 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 53.69, Last appended by leader: 69, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:54.794716  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 54 to 56, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 56 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:54.889271  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 56 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 54 index: 70. Preceding OpId from leader: term: 56 index: 70. (term mismatch)
I20250114 03:11:54.889375  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 69
I20250114 03:11:54.889392  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 54.70
I20250114 03:11:54.889411  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 56 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:54.889648  4556 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250114 03:11:54.890967  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 56 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 52 index: 68. Preceding OpId from leader: term: 56 index: 70. (index mismatch)
I20250114 03:11:54.891165  4532 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20250114 03:11:55.193974  4560 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 56 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:55.194079  4560 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 56 FOLLOWER]: Advancing to term 57
I20250114 03:11:55.194806  4560 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.194983  4560 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 57 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:55.195253  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:55.195238  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:55.195494  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:55.195595  4560 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20250114 03:11:55.203267  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 57 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 56. Current term is 57. Ops: []
I20250114 03:11:55.203549  4559 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 56.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250114 03:11:55.203675  4559 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 56 LEADER]: Stepping down as leader of term 56
I20250114 03:11:55.203711  4559 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:55.203761  4559 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 56.70, Last appended by leader: 70, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.203841  4559 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 56 FOLLOWER]: Advancing to term 57
I20250114 03:11:55.262164  4555 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 56 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:55.262265  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 56 FOLLOWER]: Advancing to term 57
I20250114 03:11:55.262929  4555 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.263077  4555 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 57 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:55.263322  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:55.263326  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:55.263403  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 57: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:55.264009  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 57.
I20250114 03:11:55.264182  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:55.264266  4555 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 57 FOLLOWER]: Leader election won for term 57
I20250114 03:11:55.264401  4555 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 57 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:55.264456  4555 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 56.70, Last appended by leader: 70, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.264948  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 56 to 57, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 57 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:55.300051  4560 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:55.300150  4560 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 57 FOLLOWER]: Advancing to term 58
I20250114 03:11:55.300841  4560 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.301007  4560 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 58 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:55.301234  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 58 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:55.301234  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 58 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:55.301334  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 57 FOLLOWER]: Advancing to term 58
I20250114 03:11:55.301851  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 58.
I20250114 03:11:55.302037  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:55.302127  4560 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 58 FOLLOWER]: Leader election won for term 58
I20250114 03:11:55.302174  4560 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 58 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:55.302227  4560 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 56.70, Last appended by leader: 70, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.302680  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 57 to 58, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 58 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:55.354476  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 58 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 57. Current term is 58. Ops: []
I20250114 03:11:55.354830  4555 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250114 03:11:55.354923  4555 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 58.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 70
I20250114 03:11:55.354967  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 57 LEADER]: Stepping down as leader of term 57
I20250114 03:11:55.355041  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 57 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:55.355108  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 57.71, Last appended by leader: 71, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.355223  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 57 FOLLOWER]: Advancing to term 58
I20250114 03:11:55.384074  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 58 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 57 index: 71. Preceding OpId from leader: term: 58 index: 71. (term mismatch)
I20250114 03:11:55.384145  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 70
I20250114 03:11:55.384158  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 57.71
I20250114 03:11:55.384173  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:55.384363  4560 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250114 03:11:55.385362  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 58 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 56 index: 70. Preceding OpId from leader: term: 58 index: 71. (index mismatch)
I20250114 03:11:55.385746  4562 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250114 03:11:55.486879  4559 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 58 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:55.486982  4559 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 58 FOLLOWER]: Advancing to term 59
I20250114 03:11:55.487735  4559 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.487902  4559 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 59 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:55.488131  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 59 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:55.488131  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 59 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:55.488446  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 59 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:55.488574  4559 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 59 FOLLOWER]: Leader election lost for term 59. Reason: could not achieve majority
I20250114 03:11:55.508239  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 59 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 58. Current term is 59. Ops: []
I20250114 03:11:55.508548  4560 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 58.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250114 03:11:55.508678  4562 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 58 LEADER]: Stepping down as leader of term 58
I20250114 03:11:55.508737  4562 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:55.508807  4562 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 58.71, Last appended by leader: 71, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.508896  4562 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 58 FOLLOWER]: Advancing to term 59
I20250114 03:11:55.580842  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 58 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:55.580941  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 58 FOLLOWER]: Advancing to term 59
I20250114 03:11:55.581674  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.581852  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 59 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:55.582087  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 59 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:55.582087  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 59 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:55.582222  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 59: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:55.582834  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 59.
I20250114 03:11:55.583020  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:55.583123  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 59 FOLLOWER]: Leader election won for term 59
I20250114 03:11:55.583194  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 59 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:55.583253  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 58.71, Last appended by leader: 71, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.583739  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 58 to 59, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 59 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:55.626010  4559 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 59 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:55.626119  4559 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 59 FOLLOWER]: Advancing to term 60
I20250114 03:11:55.626811  4559 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.626996  4559 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 60 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:55.627208  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 60 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:55.627218  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 60 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:55.627331  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 59 FOLLOWER]: Advancing to term 60
I20250114 03:11:55.628094  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 60.
I20250114 03:11:55.628295  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:55.628427  4559 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 60 FOLLOWER]: Leader election won for term 60
I20250114 03:11:55.628499  4559 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 60 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:55.628584  4559 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 58.71, Last appended by leader: 71, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.629081  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 59 to 60, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 60 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:55.662179  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 60 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 59. Current term is 60. Ops: []
I20250114 03:11:55.662500  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250114 03:11:55.662647  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 59 LEADER]: Stepping down as leader of term 59
I20250114 03:11:55.662686  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:55.662757  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 59.72, Last appended by leader: 72, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.662837  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 59 FOLLOWER]: Advancing to term 60
I20250114 03:11:55.725706  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 60 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 59 index: 72. Preceding OpId from leader: term: 60 index: 72. (term mismatch)
I20250114 03:11:55.725791  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 71
I20250114 03:11:55.725805  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 59.72
I20250114 03:11:55.725822  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 60 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:55.726020  4559 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250114 03:11:55.727000  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 60 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 58 index: 71. Preceding OpId from leader: term: 60 index: 72. (index mismatch)
I20250114 03:11:55.727286  4563 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250114 03:11:55.760047  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:55.828775  4562 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 60 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:55.828881  4562 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 60 FOLLOWER]: Advancing to term 61
I20250114 03:11:55.829766  4562 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.829988  4562 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 61 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:55.830276  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 61 candidate_status { last_received { term: 60 index: 72 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:55.830686  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 61 candidate_status { last_received { term: 60 index: 72 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:55.830981  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:55.831099  4562 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20250114 03:11:55.838948  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 61 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 60. Current term is 61. Ops: []
I20250114 03:11:55.839174  4559 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 60.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250114 03:11:55.839299  4563 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 60 LEADER]: Stepping down as leader of term 60
I20250114 03:11:55.839336  4563 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 60 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:55.839403  4563 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 60.72, Last appended by leader: 72, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.839481  4563 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 60 FOLLOWER]: Advancing to term 61
I20250114 03:11:55.908520  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 60 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:55.908655  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 60 FOLLOWER]: Advancing to term 61
I20250114 03:11:55.909365  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.909515  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 61 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:55.909772  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 61 candidate_status { last_received { term: 60 index: 72 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:55.909772  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 61 candidate_status { last_received { term: 60 index: 72 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:55.909933  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 61: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:55.910499  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 61.
I20250114 03:11:55.910676  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:55.910756  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 61 FOLLOWER]: Leader election won for term 61
I20250114 03:11:55.910832  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 61 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:55.910889  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 60.72, Last appended by leader: 72, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.911319  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 60 to 61, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 61 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:55.953532  4562 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 61 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:55.953637  4562 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 61 FOLLOWER]: Advancing to term 62
I20250114 03:11:55.954371  4562 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.954543  4562 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 62 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:55.954771  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 62 candidate_status { last_received { term: 60 index: 72 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:55.954771  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 62 candidate_status { last_received { term: 60 index: 72 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:55.954897  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 61 FOLLOWER]: Advancing to term 62
I20250114 03:11:55.955493  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 62.
I20250114 03:11:55.955684  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 62 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:55.955801  4562 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 62 FOLLOWER]: Leader election won for term 62
I20250114 03:11:55.955873  4562 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 62 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:55.955937  4562 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 60.72, Last appended by leader: 72, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:55.956403  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 61 to 62, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 62 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:56.022778  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 62 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 61. Current term is 62. Ops: []
I20250114 03:11:56.023092  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250114 03:11:56.023209  4561 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 62.73 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 72
I20250114 03:11:56.023248  4555 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 61 LEADER]: Stepping down as leader of term 61
I20250114 03:11:56.023291  4555 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 61 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:56.023355  4555 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 61.73, Last appended by leader: 73, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.023461  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 61 FOLLOWER]: Advancing to term 62
I20250114 03:11:56.046787  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 62 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 60 index: 72. Preceding OpId from leader: term: 62 index: 73. (index mismatch)
I20250114 03:11:56.047084  4562 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250114 03:11:56.048131  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 62 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 61 index: 73. Preceding OpId from leader: term: 62 index: 73. (term mismatch)
I20250114 03:11:56.048187  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 72
I20250114 03:11:56.048208  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 61.73
I20250114 03:11:56.048231  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 62 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:56.048424  4560 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250114 03:11:56.232223  4563 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 62 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:56.232313  4563 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 62 FOLLOWER]: Advancing to term 63
I20250114 03:11:56.233014  4563 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.233186  4563 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 63 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:56.233372  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 63 candidate_status { last_received { term: 62 index: 73 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:56.233392  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 63 candidate_status { last_received { term: 62 index: 73 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:56.233584  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 63 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:56.233696  4563 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 63 FOLLOWER]: Leader election lost for term 63. Reason: could not achieve majority
I20250114 03:11:56.248163  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 63 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 62. Current term is 63. Ops: []
I20250114 03:11:56.248517  4560 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 62.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250114 03:11:56.248682  4565 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 62 LEADER]: Stepping down as leader of term 62
I20250114 03:11:56.248737  4565 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 62 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:56.248819  4565 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 62.73, Last appended by leader: 73, Current term: 62, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.248920  4565 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 62 FOLLOWER]: Advancing to term 63
I20250114 03:11:56.329010  4555 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 62 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:56.329130  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 62 FOLLOWER]: Advancing to term 63
I20250114 03:11:56.329954  4555 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.330152  4555 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 63 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:56.330421  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 63 candidate_status { last_received { term: 62 index: 73 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:56.330421  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 63 candidate_status { last_received { term: 62 index: 73 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:56.330577  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 63: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:56.331187  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 63.
I20250114 03:11:56.331398  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:56.331511  4555 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 63 FOLLOWER]: Leader election won for term 63
I20250114 03:11:56.331590  4555 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 63 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:56.331651  4555 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 62.73, Last appended by leader: 73, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.332206  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 62 to 63, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 63 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:56.362092  4563 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 63 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:56.362254  4563 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 63 FOLLOWER]: Advancing to term 64
I20250114 03:11:56.362983  4563 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.363161  4563 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 64 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:56.363387  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 64 candidate_status { last_received { term: 62 index: 73 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:56.363420  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 64 candidate_status { last_received { term: 62 index: 73 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:56.363497  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 63 FOLLOWER]: Advancing to term 64
I20250114 03:11:56.364113  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 64.
I20250114 03:11:56.364279  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:56.364377  4563 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 64 FOLLOWER]: Leader election won for term 64
I20250114 03:11:56.364478  4563 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 64 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:56.364523  4563 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 62.73, Last appended by leader: 73, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.364991  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 63 to 64, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 64 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:56.422287  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 64 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 63. Current term is 64. Ops: []
I20250114 03:11:56.422602  4555 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250114 03:11:56.422739  4555 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 64.74 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 73
I20250114 03:11:56.422784  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 63 LEADER]: Stepping down as leader of term 63
I20250114 03:11:56.422824  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 63 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:56.422874  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 63.74, Last appended by leader: 74, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.422956  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 63 FOLLOWER]: Advancing to term 64
I20250114 03:11:56.447575  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 64 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 62 index: 73. Preceding OpId from leader: term: 64 index: 74. (index mismatch)
I20250114 03:11:56.447880  4563 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250114 03:11:56.448868  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 64 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 63 index: 74. Preceding OpId from leader: term: 64 index: 74. (term mismatch)
I20250114 03:11:56.448927  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 73
I20250114 03:11:56.448951  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 63.74
I20250114 03:11:56.448972  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 64 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:56.449188  4563 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250114 03:11:56.549829  4565 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 64 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:56.549960  4565 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 64 FOLLOWER]: Advancing to term 65
I20250114 03:11:56.550653  4565 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.550838  4565 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 65 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:56.551065  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 65 candidate_status { last_received { term: 64 index: 74 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:56.551065  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 65 candidate_status { last_received { term: 64 index: 74 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:56.551337  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 65 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:56.551443  4565 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 65 FOLLOWER]: Leader election lost for term 65. Reason: could not achieve majority
I20250114 03:11:56.571944  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 65 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 64. Current term is 65. Ops: []
I20250114 03:11:56.572253  4567 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 64.74, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250114 03:11:56.572396  4566 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 64 LEADER]: Stepping down as leader of term 64
I20250114 03:11:56.572448  4566 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 64 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:56.572551  4566 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 64.74, Last appended by leader: 74, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.572630  4566 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 64 FOLLOWER]: Advancing to term 65
I20250114 03:11:56.648553  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 64 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:56.648654  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 64 FOLLOWER]: Advancing to term 65
I20250114 03:11:56.649351  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.649531  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 65 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:56.649755  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 65 candidate_status { last_received { term: 64 index: 74 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:56.649760  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 65 candidate_status { last_received { term: 64 index: 74 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:56.649935  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 65 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 65: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:56.650441  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 65.
I20250114 03:11:56.650609  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:56.650696  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 65 FOLLOWER]: Leader election won for term 65
I20250114 03:11:56.650753  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 65 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:56.650807  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 64.74, Last appended by leader: 74, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.651353  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 64 to 65, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 65 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:56.675894  4565 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 65 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:56.675994  4565 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 65 FOLLOWER]: Advancing to term 66
I20250114 03:11:56.676684  4565 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.676868  4565 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 66 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:56.677093  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 66 candidate_status { last_received { term: 64 index: 74 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:56.677155  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 65 FOLLOWER]: Advancing to term 66
I20250114 03:11:56.677115  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 66 candidate_status { last_received { term: 64 index: 74 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:56.677677  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 66 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 66.
I20250114 03:11:56.677884  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 66 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:56.677995  4565 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 66 FOLLOWER]: Leader election won for term 66
I20250114 03:11:56.678053  4565 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 66 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:56.678112  4565 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 64.74, Last appended by leader: 74, Current term: 66, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.678571  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 65 to 66, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 66 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:56.733021  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 66 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 65. Current term is 66. Ops: []
I20250114 03:11:56.733305  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250114 03:11:56.733398  4561 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 66.75 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 74
I20250114 03:11:56.733428  4555 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 65 LEADER]: Stepping down as leader of term 65
I20250114 03:11:56.733469  4555 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 65 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:56.733525  4555 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 65.75, Last appended by leader: 75, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.733620  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 65 FOLLOWER]: Advancing to term 66
I20250114 03:11:56.760288  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:56.793123  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 66 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 65 index: 75. Preceding OpId from leader: term: 66 index: 75. (term mismatch)
I20250114 03:11:56.793208  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 74
I20250114 03:11:56.793236  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 65.75
I20250114 03:11:56.793262  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 66 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:56.793512  4565 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250114 03:11:56.794636  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 66 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 64 index: 74. Preceding OpId from leader: term: 66 index: 75. (index mismatch)
I20250114 03:11:56.794853  4560 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250114 03:11:56.896210  4566 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 66 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:56.896349  4566 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 66 FOLLOWER]: Advancing to term 67
I20250114 03:11:56.897994  4566 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.898280  4566 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 67 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:56.898344  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 67 candidate_status { last_received { term: 66 index: 75 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:56.898483  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 67 candidate_status { last_received { term: 66 index: 75 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:56.898814  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 67 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:56.898948  4566 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 67 FOLLOWER]: Leader election lost for term 67. Reason: could not achieve majority
I20250114 03:11:56.907884  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 67 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 66. Current term is 67. Ops: []
I20250114 03:11:56.908106  4568 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 66.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250114 03:11:56.908210  4568 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 66 LEADER]: Stepping down as leader of term 66
I20250114 03:11:56.908241  4568 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 66 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:56.908303  4568 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 66.75, Last appended by leader: 75, Current term: 66, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.908385  4568 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 66 FOLLOWER]: Advancing to term 67
I20250114 03:11:56.995749  4555 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 66 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:56.995864  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 66 FOLLOWER]: Advancing to term 67
I20250114 03:11:56.996542  4555 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.996712  4555 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 67 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:56.996994  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 67 candidate_status { last_received { term: 66 index: 75 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:56.996994  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 67 candidate_status { last_received { term: 66 index: 75 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:56.997251  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 67 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 67: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:56.997817  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 67.
I20250114 03:11:56.998009  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 67 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:56.998114  4555 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 67 FOLLOWER]: Leader election won for term 67
I20250114 03:11:56.998175  4555 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 67 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:56.998234  4555 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 66.75, Last appended by leader: 75, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:56.998693  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 66 to 67, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 67 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:57.001529  4566 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 67 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:57.001582  4566 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 67 FOLLOWER]: Advancing to term 68
I20250114 03:11:57.002234  4566 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.002357  4566 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 68 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:57.002511  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 68 candidate_status { last_received { term: 66 index: 75 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:57.002512  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 68 candidate_status { last_received { term: 66 index: 75 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:57.002609  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 67 FOLLOWER]: Advancing to term 68
I20250114 03:11:57.003114  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 68.
I20250114 03:11:57.003273  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 68 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:57.003360  4566 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 68 FOLLOWER]: Leader election won for term 68
I20250114 03:11:57.003409  4566 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 68 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:57.003446  4566 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 66.75, Last appended by leader: 75, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.003872  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 67 to 68, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 68 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:57.085283  4425 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 67 LEADER]: Stepping down as leader of term 67
I20250114 03:11:57.085371  4425 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 67 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:57.085439  4425 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 67.76, Last appended by leader: 76, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.085556  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 67 FOLLOWER]: Advancing to term 68
I20250114 03:11:57.086293  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 68 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 67 index: 76. Preceding OpId from leader: term: 68 index: 76. (term mismatch)
I20250114 03:11:57.086338  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 75
I20250114 03:11:57.086359  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 67.76
I20250114 03:11:57.086385  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 68 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:57.086683  4566 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250114 03:11:57.087834  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 68 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 66 index: 75. Preceding OpId from leader: term: 68 index: 76. (index mismatch)
I20250114 03:11:57.088112  4566 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250114 03:11:57.274242  4568 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 68 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:57.274345  4568 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 68 FOLLOWER]: Advancing to term 69
I20250114 03:11:57.275161  4568 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.275295  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 69 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 68. Current term is 69. Ops: []
I20250114 03:11:57.275456  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:57.275496  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:57.275595  4569 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 68.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250114 03:11:57.275691  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 69 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:57.275730  4569 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 68 LEADER]: Stepping down as leader of term 68
I20250114 03:11:57.275347  4568 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 69 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:57.275792  4565 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 69 FOLLOWER]: Leader election lost for term 69. Reason: could not achieve majority
I20250114 03:11:57.275801  4569 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 68 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:57.275949  4569 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 68.76, Last appended by leader: 76, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.276031  4569 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 68 FOLLOWER]: Advancing to term 69
I20250114 03:11:57.374181  4555 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 68 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:57.374285  4555 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 68 FOLLOWER]: Advancing to term 69
I20250114 03:11:57.374974  4555 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.375147  4555 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 69 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:57.375398  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:57.375416  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:57.375510  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 69: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:57.376034  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 69.
I20250114 03:11:57.376224  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:57.376360  4555 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 69 FOLLOWER]: Leader election won for term 69
I20250114 03:11:57.376441  4555 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 69 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:57.376543  4555 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 68.76, Last appended by leader: 76, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.377112  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 68 to 69, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 69 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:57.407909  4565 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:57.408025  4565 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 69 FOLLOWER]: Advancing to term 70
I20250114 03:11:57.408732  4565 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.408910  4565 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 70 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:57.409142  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 70 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:57.409204  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 70 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:57.409227  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 69 FOLLOWER]: Advancing to term 70
I20250114 03:11:57.409924  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 70.
I20250114 03:11:57.410089  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:57.410178  4565 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 70 FOLLOWER]: Leader election won for term 70
I20250114 03:11:57.410228  4565 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 70 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:57.410272  4565 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 68.76, Last appended by leader: 76, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.410717  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 69 to 70, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 70 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:57.458508  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 70 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 69. Current term is 70. Ops: []
I20250114 03:11:57.458818  4555 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250114 03:11:57.458938  4555 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 70.77 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 76
I20250114 03:11:57.458977  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 69 LEADER]: Stepping down as leader of term 69
I20250114 03:11:57.459026  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 69 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:57.459092  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 69.77, Last appended by leader: 77, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.459187  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 69 FOLLOWER]: Advancing to term 70
I20250114 03:11:57.486232  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 70 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 68 index: 76. Preceding OpId from leader: term: 70 index: 77. (index mismatch)
I20250114 03:11:57.486550  4565 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250114 03:11:57.487747  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 70 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 69 index: 77. Preceding OpId from leader: term: 70 index: 77. (term mismatch)
I20250114 03:11:57.487829  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 76
I20250114 03:11:57.487854  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 69.77
I20250114 03:11:57.487882  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 70 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:57.488050  4571 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250114 03:11:57.587881  4569 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 70 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:57.587987  4569 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 70 FOLLOWER]: Advancing to term 71
I20250114 03:11:57.588699  4569 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.588907  4569 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 71 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:57.589140  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 71 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:57.589208  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 70 FOLLOWER]: Advancing to term 71
I20250114 03:11:57.589151  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 71 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:57.589809  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 71.
I20250114 03:11:57.590039  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 71 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:57.590139  4569 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 71 FOLLOWER]: Leader election won for term 71
I20250114 03:11:57.590313  4569 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 71 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:57.590371  4569 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 70.77, Last appended by leader: 77, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.590893  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 70 to 71, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 71 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:57.592803  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 71 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 70. Current term is 71. Ops: []
I20250114 03:11:57.592978  4570 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 70.77, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250114 03:11:57.593083  4570 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 70 LEADER]: Stepping down as leader of term 70
I20250114 03:11:57.593118  4570 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 70 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:57.593173  4570 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 70.77, Last appended by leader: 77, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.593255  4570 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 70 FOLLOWER]: Advancing to term 71
I20250114 03:11:57.686451  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 71 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 70 index: 77. Preceding OpId from leader: term: 71 index: 78. (index mismatch)
I20250114 03:11:57.686757  4569 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250114 03:11:57.687829  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 71 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 70 index: 77. Preceding OpId from leader: term: 71 index: 78. (index mismatch)
I20250114 03:11:57.688005  4567 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250114 03:11:57.760524  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:57.789189  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 71 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:57.789296  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 71 FOLLOWER]: Advancing to term 72
I20250114 03:11:57.789997  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.790155  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 72 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:57.790400  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 72 candidate_status { last_received { term: 71 index: 78 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:57.790401  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 72 candidate_status { last_received { term: 71 index: 78 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:57.790534  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 71 FOLLOWER]: Advancing to term 72
I20250114 03:11:57.791091  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 72.
I20250114 03:11:57.791301  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 72 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:57.791392  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 72 FOLLOWER]: Leader election won for term 72
I20250114 03:11:57.791445  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 72 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:57.791479  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 71.78, Last appended by leader: 78, Current term: 72, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.791967  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 71 to 72, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 72 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:57.792161  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 72 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 71. Current term is 72. Ops: []
I20250114 03:11:57.792419  4567 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 71.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250114 03:11:57.792483  4567 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 72.79 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 78
I20250114 03:11:57.792600  4572 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 71 LEADER]: Stepping down as leader of term 71
I20250114 03:11:57.792656  4572 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 71 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:57.792749  4572 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 71.78, Last appended by leader: 78, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:57.792842  4572 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 71 FOLLOWER]: Advancing to term 72
I20250114 03:11:57.893594  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 72 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 71 index: 78. Preceding OpId from leader: term: 72 index: 79. (index mismatch)
I20250114 03:11:57.893976  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250114 03:11:57.895143  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 72 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 71 index: 78. Preceding OpId from leader: term: 72 index: 79. (index mismatch)
I20250114 03:11:57.895345  4573 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250114 03:11:58.018352  4572 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 72 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:58.018489  4572 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 72 FOLLOWER]: Advancing to term 73
I20250114 03:11:58.019191  4572 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.019351  4572 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 73 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:58.019582  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 73 candidate_status { last_received { term: 72 index: 79 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:58.019620  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 73 candidate_status { last_received { term: 72 index: 79 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:58.019831  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 73 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:58.019927  4572 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 73 FOLLOWER]: Leader election lost for term 73. Reason: could not achieve majority
I20250114 03:11:58.020067  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 73 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 72. Current term is 73. Ops: []
I20250114 03:11:58.020262  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 72.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250114 03:11:58.020362  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 72 LEADER]: Stepping down as leader of term 72
I20250114 03:11:58.020395  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 72 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:58.020447  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 72.79, Last appended by leader: 79, Current term: 72, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.020527  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 72 FOLLOWER]: Advancing to term 73
I20250114 03:11:58.074870  4570 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 72 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:58.074976  4570 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 72 FOLLOWER]: Advancing to term 73
I20250114 03:11:58.075701  4570 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.075876  4570 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 73 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:58.076133  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 73 candidate_status { last_received { term: 72 index: 79 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:58.076151  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 73 candidate_status { last_received { term: 72 index: 79 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:58.076251  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 73: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:11:58.076750  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 73.
I20250114 03:11:58.076954  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:58.077064  4570 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 73 FOLLOWER]: Leader election won for term 73
I20250114 03:11:58.077138  4570 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 73 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:58.077221  4570 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 72.79, Last appended by leader: 79, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.077670  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 72 to 73, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 73 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:58.155289  4572 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:58.155404  4572 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 73 FOLLOWER]: Advancing to term 74
I20250114 03:11:58.156095  4572 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.156267  4572 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 74 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:58.156478  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 74 candidate_status { last_received { term: 72 index: 79 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:58.156486  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 74 candidate_status { last_received { term: 72 index: 79 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:58.156610  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 73 FOLLOWER]: Advancing to term 74
I20250114 03:11:58.157196  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 74.
I20250114 03:11:58.157366  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 74 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:58.157455  4572 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 74 FOLLOWER]: Leader election won for term 74
I20250114 03:11:58.157516  4572 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 74 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:58.157577  4572 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 72.79, Last appended by leader: 79, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.158186  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 73 to 74, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 74 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:58.185292  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 74 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 73. Current term is 74. Ops: []
I20250114 03:11:58.185591  4570 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250114 03:11:58.185708  4570 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 74.80 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 79
I20250114 03:11:58.185750  4571 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 73 LEADER]: Stepping down as leader of term 73
I20250114 03:11:58.185804  4571 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 73 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:58.185895  4571 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 73.80, Last appended by leader: 80, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.185983  4571 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 73 FOLLOWER]: Advancing to term 74
I20250114 03:11:58.263950  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 74 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 73 index: 80. Preceding OpId from leader: term: 74 index: 80. (term mismatch)
I20250114 03:11:58.264050  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 79
I20250114 03:11:58.264083  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 73.80
I20250114 03:11:58.264154  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 74 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:58.264430  4572 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250114 03:11:58.265702  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 74 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 72 index: 79. Preceding OpId from leader: term: 74 index: 80. (index mismatch)
I20250114 03:11:58.265914  4572 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250114 03:11:58.366448  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 74 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:58.366566  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 74 FOLLOWER]: Advancing to term 75
I20250114 03:11:58.367255  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.367432  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 75 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:58.367637  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 75 candidate_status { last_received { term: 74 index: 80 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:58.367643  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 75 candidate_status { last_received { term: 74 index: 80 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:58.367748  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 74 FOLLOWER]: Advancing to term 75
I20250114 03:11:58.368294  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 75 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 75.
I20250114 03:11:58.368474  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 75 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:58.368557  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 75 FOLLOWER]: Leader election won for term 75
I20250114 03:11:58.368628  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 75 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:58.368678  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 74.80, Last appended by leader: 80, Current term: 75, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.369110  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 74 to 75, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 75 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:58.371171  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 75 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 74. Current term is 75. Ops: []
I20250114 03:11:58.371387  4567 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 74.80, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250114 03:11:58.371465  4567 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 75.81 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 80
I20250114 03:11:58.371501  4567 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 74 LEADER]: Stepping down as leader of term 74
I20250114 03:11:58.371529  4567 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 74 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:58.371600  4567 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 74.80, Last appended by leader: 80, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.371685  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 74 FOLLOWER]: Advancing to term 75
I20250114 03:11:58.479770  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 75 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 74 index: 80. Preceding OpId from leader: term: 75 index: 81. (index mismatch)
I20250114 03:11:58.480089  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250114 03:11:58.481009  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 75 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 74 index: 80. Preceding OpId from leader: term: 75 index: 81. (index mismatch)
I20250114 03:11:58.481235  4574 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250114 03:11:58.581580  4571 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 75 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:58.581689  4571 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 75 FOLLOWER]: Advancing to term 76
I20250114 03:11:58.582463  4571 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.582661  4571 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 76 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:58.582863  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 76 candidate_status { last_received { term: 75 index: 81 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:58.582863  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 76 candidate_status { last_received { term: 75 index: 81 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:58.583134  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 76 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:58.583240  4571 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 76 FOLLOWER]: Leader election lost for term 76. Reason: could not achieve majority
I20250114 03:11:58.598100  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 76 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 75. Current term is 76. Ops: []
I20250114 03:11:58.598363  4573 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 75.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250114 03:11:58.598500  4573 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 75 LEADER]: Stepping down as leader of term 75
I20250114 03:11:58.598538  4573 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 75 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:58.598604  4573 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 75.81, Last appended by leader: 81, Current term: 75, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.598685  4573 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 75 FOLLOWER]: Advancing to term 76
I20250114 03:11:58.665434  4567 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 75 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:58.665552  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 75 FOLLOWER]: Advancing to term 76
I20250114 03:11:58.666236  4567 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.666432  4567 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 76 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:58.666679  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 76 candidate_status { last_received { term: 75 index: 81 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:58.666759  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 76: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:58.666679  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 76 candidate_status { last_received { term: 75 index: 81 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:58.667368  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 76.
I20250114 03:11:58.667550  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 76 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:58.667639  4567 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 76 FOLLOWER]: Leader election won for term 76
I20250114 03:11:58.667703  4567 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 76 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:58.667766  4567 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 75.81, Last appended by leader: 81, Current term: 76, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.668319  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 75 to 76, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 76 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:58.730679  4571 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 76 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:58.730808  4571 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 76 FOLLOWER]: Advancing to term 77
I20250114 03:11:58.731521  4571 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.731747  4571 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 77 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:58.731976  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 77 candidate_status { last_received { term: 75 index: 81 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:58.732040  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 76 FOLLOWER]: Advancing to term 77
I20250114 03:11:58.731983  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 77 candidate_status { last_received { term: 75 index: 81 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:58.732646  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 77.
I20250114 03:11:58.732827  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 77 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:58.732923  4571 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 77 FOLLOWER]: Leader election won for term 77
I20250114 03:11:58.733043  4571 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 77 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:58.733095  4571 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 75.81, Last appended by leader: 81, Current term: 77, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.733537  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 76 to 77, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 77 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:58.752651  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 77 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 76. Current term is 77. Ops: []
I20250114 03:11:58.752928  4567 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250114 03:11:58.753042  4567 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 76 LEADER]: Stepping down as leader of term 76
I20250114 03:11:58.753087  4567 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 76 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:58.753145  4567 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 76.82, Last appended by leader: 82, Current term: 76, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.753243  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 76 FOLLOWER]: Advancing to term 77
I20250114 03:11:58.766654  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:58.817078  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 77 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 76 index: 82. Preceding OpId from leader: term: 77 index: 82. (term mismatch)
I20250114 03:11:58.817157  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 81
I20250114 03:11:58.817170  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 76.82
I20250114 03:11:58.817186  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 77 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:11:58.817405  4571 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250114 03:11:58.818493  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 77 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 75 index: 81. Preceding OpId from leader: term: 77 index: 82. (index mismatch)
I20250114 03:11:58.818696  4571 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250114 03:11:58.919437  4573 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 77 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:11:58.919539  4573 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 77 FOLLOWER]: Advancing to term 78
I20250114 03:11:58.920431  4573 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.920854  4573 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 78 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:58.920900  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 78 candidate_status { last_received { term: 77 index: 82 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:58.921010  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 78 candidate_status { last_received { term: 77 index: 82 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:58.921084  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 77 FOLLOWER]: Advancing to term 78
I20250114 03:11:58.921629  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 78.
I20250114 03:11:58.921800  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 78 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:58.921886  4573 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 78 FOLLOWER]: Leader election won for term 78
I20250114 03:11:58.921960  4573 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 78 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:58.922014  4573 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 77.82, Last appended by leader: 82, Current term: 78, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.922461  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 77 to 78, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 78 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:58.932806  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 78 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 77. Current term is 78. Ops: []
I20250114 03:11:58.933137  4571 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 77.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250114 03:11:58.933275  4571 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 78.83 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 82
I20250114 03:11:58.933315  4575 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 77 LEADER]: Stepping down as leader of term 77
I20250114 03:11:58.933351  4575 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 77 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:58.933472  4575 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 77.82, Last appended by leader: 82, Current term: 77, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:58.933607  4575 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 77 FOLLOWER]: Advancing to term 78
I20250114 03:11:59.009034  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 78 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 77 index: 82. Preceding OpId from leader: term: 78 index: 83. (index mismatch)
I20250114 03:11:59.009368  4573 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250114 03:11:59.010375  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 78 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 77 index: 82. Preceding OpId from leader: term: 78 index: 83. (index mismatch)
I20250114 03:11:59.010618  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250114 03:11:59.111336  4567 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 78 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:59.111449  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 78 FOLLOWER]: Advancing to term 79
I20250114 03:11:59.112147  4567 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.112310  4567 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 79 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:59.112540  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 79 candidate_status { last_received { term: 78 index: 83 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:59.112547  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 79 candidate_status { last_received { term: 78 index: 83 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:59.112649  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 78 FOLLOWER]: Advancing to term 79
I20250114 03:11:59.113191  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 79.
I20250114 03:11:59.113421  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 79 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:59.113505  4567 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 79 FOLLOWER]: Leader election won for term 79
I20250114 03:11:59.113574  4567 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 79 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:59.113612  4567 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 78.83, Last appended by leader: 83, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.114078  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 78 to 79, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 79 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:59.115082  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 79 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 78. Current term is 79. Ops: []
I20250114 03:11:59.115262  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 78.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250114 03:11:59.115358  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 78 LEADER]: Stepping down as leader of term 78
I20250114 03:11:59.115411  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 78 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:59.115473  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 78.83, Last appended by leader: 83, Current term: 78, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.115550  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 78 FOLLOWER]: Advancing to term 79
I20250114 03:11:59.208518  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 79 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 78 index: 83. Preceding OpId from leader: term: 79 index: 84. (index mismatch)
I20250114 03:11:59.208890  4567 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250114 03:11:59.215031  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 79 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 78 index: 83. Preceding OpId from leader: term: 79 index: 84. (index mismatch)
I20250114 03:11:59.215312  4572 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250114 03:11:59.405190  4575 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 79 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:59.405293  4575 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 79 FOLLOWER]: Advancing to term 80
I20250114 03:11:59.406018  4575 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.406173  4575 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 80 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:59.406411  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 80 candidate_status { last_received { term: 79 index: 84 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:59.406416  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 80 candidate_status { last_received { term: 79 index: 84 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:59.406663  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 80 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:59.406775  4575 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 80 FOLLOWER]: Leader election lost for term 80. Reason: could not achieve majority
I20250114 03:11:59.429448  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 80 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 79. Current term is 80. Ops: []
I20250114 03:11:59.429721  4576 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 79.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250114 03:11:59.429876  4567 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 79 LEADER]: Stepping down as leader of term 79
I20250114 03:11:59.429932  4567 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 79 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:59.429986  4567 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 79.84, Last appended by leader: 84, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.430078  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 79 FOLLOWER]: Advancing to term 80
I20250114 03:11:59.432883  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 79 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:11:59.432950  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 79 FOLLOWER]: Advancing to term 80
I20250114 03:11:59.433458  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.433601  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 80 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:59.433773  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 80 candidate_status { last_received { term: 79 index: 84 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:59.433840  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 80 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 80: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:59.433799  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 80 candidate_status { last_received { term: 79 index: 84 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:59.434381  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 80.
I20250114 03:11:59.434551  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:59.434635  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 80 FOLLOWER]: Leader election won for term 80
I20250114 03:11:59.434693  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 80 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:59.434748  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 79.84, Last appended by leader: 84, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.435153  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 79 to 80, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 80 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:11:59.523800  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 80 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 79 index: 84. Preceding OpId from leader: term: 80 index: 85. (index mismatch)
I20250114 03:11:59.524075  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250114 03:11:59.525206  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 80 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 79 index: 84. Preceding OpId from leader: term: 80 index: 85. (index mismatch)
I20250114 03:11:59.525447  4577 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250114 03:11:59.766930  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:11:59.874986  4578 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 80 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:59.875097  4578 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 80 FOLLOWER]: Advancing to term 81
I20250114 03:11:59.875928  4578 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.876117  4578 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 81 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:59.876348  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:59.876348  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:59.876582  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:11:59.876694  4578 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20250114 03:11:59.893755  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 81 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 80. Current term is 81. Ops: []
I20250114 03:11:59.894037  4573 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 80.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250114 03:11:59.894148  4573 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 80 LEADER]: Stepping down as leader of term 80
I20250114 03:11:59.894181  4573 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 80 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:11:59.894243  4573 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.894315  4573 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 80 FOLLOWER]: Advancing to term 81
I20250114 03:11:59.908517  4567 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 80 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:11:59.908614  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 80 FOLLOWER]: Advancing to term 81
I20250114 03:11:59.909305  4567 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.909475  4567 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 81 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:11:59.909701  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:11:59.909775  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 81: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:11:59.909713  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:59.910449  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 81.
I20250114 03:11:59.910677  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 81 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:11:59.910779  4567 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 81 FOLLOWER]: Leader election won for term 81
I20250114 03:11:59.910828  4567 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 81 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:11:59.910867  4567 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.911312  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 80 to 81, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 81 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:11:59.984305  4578 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 81 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:11:59.984414  4578 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 81 FOLLOWER]: Advancing to term 82
I20250114 03:11:59.985129  4578 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.985275  4578 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 82 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:11:59.985488  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 82 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:11:59.985561  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 81 FOLLOWER]: Advancing to term 82
I20250114 03:11:59.985498  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 82 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:11:59.986184  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 82.
I20250114 03:11:59.986361  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 82 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:11:59.986457  4578 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 82 FOLLOWER]: Leader election won for term 82
I20250114 03:11:59.986526  4578 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 82 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:11:59.986588  4578 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:11:59.987005  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 81 to 82, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 82 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:00.012857  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 82 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 81. Current term is 82. Ops: []
I20250114 03:12:00.013154  4567 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250114 03:12:00.013296  4567 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 81 LEADER]: Stepping down as leader of term 81
I20250114 03:12:00.013319  4567 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 81 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:00.013382  4567 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 81.86, Last appended by leader: 86, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.013458  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 81 FOLLOWER]: Advancing to term 82
I20250114 03:12:00.074508  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 82 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 81 index: 86. Preceding OpId from leader: term: 82 index: 86. (term mismatch)
I20250114 03:12:00.074590  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 85
I20250114 03:12:00.074613  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 81.86
I20250114 03:12:00.074635  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 82 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:00.074891  4578 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250114 03:12:00.075925  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 82 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 80 index: 85. Preceding OpId from leader: term: 82 index: 86. (index mismatch)
I20250114 03:12:00.076118  4578 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250114 03:12:00.177132  4573 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 82 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:00.177232  4573 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 82 FOLLOWER]: Advancing to term 83
I20250114 03:12:00.178012  4573 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.178186  4573 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 83 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:00.178414  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:00.178414  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:00.178746  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 83 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:00.178839  4573 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 83 FOLLOWER]: Leader election lost for term 83. Reason: could not achieve majority
I20250114 03:12:00.197263  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 83 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 82. Current term is 83. Ops: []
I20250114 03:12:00.197536  4579 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 82.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250114 03:12:00.197646  4579 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 82 LEADER]: Stepping down as leader of term 82
I20250114 03:12:00.197681  4579 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 82 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:00.197746  4579 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.197830  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 82 FOLLOWER]: Advancing to term 83
I20250114 03:12:00.255273  4567 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 82 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:00.255388  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 82 FOLLOWER]: Advancing to term 83
I20250114 03:12:00.256089  4567 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.256263  4567 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 83 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:00.256484  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:00.256489  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:00.256645  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 83 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 83: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:00.257151  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 83.
I20250114 03:12:00.257313  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:00.257391  4567 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 83 FOLLOWER]: Leader election won for term 83
I20250114 03:12:00.257443  4567 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 83 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:00.257498  4567 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.257943  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 82 to 83, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 83 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:00.306802  4573 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 83 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:00.306900  4573 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 83 FOLLOWER]: Advancing to term 84
I20250114 03:12:00.307574  4573 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.307722  4573 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 84 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:00.307943  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 84 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:00.307943  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 84 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:00.308045  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 83 FOLLOWER]: Advancing to term 84
I20250114 03:12:00.308612  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 84.
I20250114 03:12:00.308779  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 84 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:00.308892  4573 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 84 FOLLOWER]: Leader election won for term 84
I20250114 03:12:00.308948  4573 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 84 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:00.308997  4573 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.309425  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 83 to 84, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 84 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:00.348389  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 84 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 83. Current term is 84. Ops: []
I20250114 03:12:00.348695  4567 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250114 03:12:00.348804  4567 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 84.87 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 86
I20250114 03:12:00.348834  4567 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 83 LEADER]: Stepping down as leader of term 83
I20250114 03:12:00.348863  4567 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 83 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:00.348922  4567 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 83.87, Last appended by leader: 87, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.349004  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 83 FOLLOWER]: Advancing to term 84
I20250114 03:12:00.420922  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 84 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 82 index: 86. Preceding OpId from leader: term: 84 index: 87. (index mismatch)
I20250114 03:12:00.421205  4573 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250114 03:12:00.422225  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 84 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 83 index: 87. Preceding OpId from leader: term: 84 index: 87. (term mismatch)
I20250114 03:12:00.422279  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 86
I20250114 03:12:00.422303  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 83.87
I20250114 03:12:00.422327  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 84 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:00.422502  4573 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250114 03:12:00.606833  4579 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 84 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:00.606935  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 84 FOLLOWER]: Advancing to term 85
I20250114 03:12:00.607648  4579 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.607821  4579 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 85 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:00.608104  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:00.608104  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:00.608421  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 85 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:00.608520  4579 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 85 FOLLOWER]: Leader election lost for term 85. Reason: could not achieve majority
I20250114 03:12:00.613807  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 85 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 84. Current term is 85. Ops: []
I20250114 03:12:00.614061  4580 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 84.87, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250114 03:12:00.614187  4580 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 84 LEADER]: Stepping down as leader of term 84
I20250114 03:12:00.614226  4580 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 84 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:00.614296  4580 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.614373  4580 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 84 FOLLOWER]: Advancing to term 85
I20250114 03:12:00.630863  4567 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 84 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:00.630952  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 84 FOLLOWER]: Advancing to term 85
I20250114 03:12:00.631579  4567 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.631724  4567 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 85 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:00.631959  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:00.631959  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:00.632081  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 85 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 85: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:00.632668  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 85.
I20250114 03:12:00.632866  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:00.632953  4567 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 85 FOLLOWER]: Leader election won for term 85
I20250114 03:12:00.633000  4567 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 85 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:00.633054  4567 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.633493  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 84 to 85, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 85 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:00.724212  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 85 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 84 index: 87. Preceding OpId from leader: term: 85 index: 88. (index mismatch)
I20250114 03:12:00.724453  4567 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250114 03:12:00.725317  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 85 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 84 index: 87. Preceding OpId from leader: term: 85 index: 88. (index mismatch)
I20250114 03:12:00.725500  4567 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250114 03:12:00.767199  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:00.826180  4579 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 85 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:00.826269  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 85 FOLLOWER]: Advancing to term 86
I20250114 03:12:00.826947  4579 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.827133  4579 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 86 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:00.827354  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 86 candidate_status { last_received { term: 85 index: 88 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:00.827370  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 86 candidate_status { last_received { term: 85 index: 88 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:00.827445  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 85 FOLLOWER]: Advancing to term 86
I20250114 03:12:00.828104  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 86 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 86.
I20250114 03:12:00.828241  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 86 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 85. Current term is 86. Ops: []
I20250114 03:12:00.828284  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 86 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:00.828387  4579 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 86 FOLLOWER]: Leader election won for term 86
I20250114 03:12:00.828476  4579 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 86 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:00.828559  4579 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 85.88, Last appended by leader: 88, Current term: 86, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.828481  4567 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 85.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250114 03:12:00.828644  4567 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 85 LEADER]: Stepping down as leader of term 85
I20250114 03:12:00.828682  4567 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 85 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:00.828725  4567 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 85.88, Last appended by leader: 88, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:00.828796  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 85 FOLLOWER]: Advancing to term 86
I20250114 03:12:00.829164  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 85 to 86, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 86 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:00.911689  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 86 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 85 index: 88. Preceding OpId from leader: term: 86 index: 89. (index mismatch)
I20250114 03:12:00.912041  4579 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250114 03:12:00.913059  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 86 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 85 index: 88. Preceding OpId from leader: term: 86 index: 89. (index mismatch)
I20250114 03:12:00.913247  4582 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250114 03:12:01.014575  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 86 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:01.014678  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 86 FOLLOWER]: Advancing to term 87
I20250114 03:12:01.015383  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.015566  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 87 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:01.015790  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 87 candidate_status { last_received { term: 86 index: 89 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:01.015874  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 86 FOLLOWER]: Advancing to term 87
I20250114 03:12:01.015801  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 87 candidate_status { last_received { term: 86 index: 89 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:01.016448  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 87.
I20250114 03:12:01.016606  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 87 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:01.016683  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 87 FOLLOWER]: Leader election won for term 87
I20250114 03:12:01.016736  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 87 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:01.016788  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 86.89, Last appended by leader: 89, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.017263  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 86 to 87, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 87 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:01.027932  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 87 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 86. Current term is 87. Ops: []
I20250114 03:12:01.028163  4582 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 86.89, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250114 03:12:01.028264  4582 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 86 LEADER]: Stepping down as leader of term 86
I20250114 03:12:01.028298  4582 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 86 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:01.028362  4582 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 86.89, Last appended by leader: 89, Current term: 86, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.028460  4582 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 86 FOLLOWER]: Advancing to term 87
I20250114 03:12:01.123832  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 87 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 86 index: 89. Preceding OpId from leader: term: 87 index: 90. (index mismatch)
I20250114 03:12:01.124094  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250114 03:12:01.125101  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 87 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 86 index: 89. Preceding OpId from leader: term: 87 index: 90. (index mismatch)
I20250114 03:12:01.125283  4580 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250114 03:12:01.225714  4567 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 87 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:01.225805  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 87 FOLLOWER]: Advancing to term 88
I20250114 03:12:01.226442  4567 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.226612  4567 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 88 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:01.226817  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 88 candidate_status { last_received { term: 87 index: 90 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:01.226822  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 88 candidate_status { last_received { term: 87 index: 90 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:01.227128  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 88 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:01.227231  4567 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 88 FOLLOWER]: Leader election lost for term 88. Reason: could not achieve majority
I20250114 03:12:01.234017  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 88 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 87. Current term is 88. Ops: []
I20250114 03:12:01.234258  4583 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 87.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250114 03:12:01.234393  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 87 LEADER]: Stepping down as leader of term 87
I20250114 03:12:01.234437  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 87 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:01.234509  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 87.90, Last appended by leader: 90, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.234596  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 87 FOLLOWER]: Advancing to term 88
I20250114 03:12:01.314260  4579 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 87 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:01.314371  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 87 FOLLOWER]: Advancing to term 88
I20250114 03:12:01.315402  4579 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.315611  4579 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 88 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:01.315832  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 88 candidate_status { last_received { term: 87 index: 90 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:01.315840  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 88 candidate_status { last_received { term: 87 index: 90 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:01.315975  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 88: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:01.316651  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 88.
I20250114 03:12:01.316833  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 88 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:01.316922  4579 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 88 FOLLOWER]: Leader election won for term 88
I20250114 03:12:01.316992  4579 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 88 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:01.317055  4579 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 87.90, Last appended by leader: 90, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.317492  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 87 to 88, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 88 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:01.358615  4567 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 88 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:01.358714  4567 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 88 FOLLOWER]: Advancing to term 89
I20250114 03:12:01.359360  4567 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.359540  4567 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 89 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:01.359767  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 89 candidate_status { last_received { term: 87 index: 90 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:01.359838  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 88 FOLLOWER]: Advancing to term 89
I20250114 03:12:01.359773  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 89 candidate_status { last_received { term: 87 index: 90 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:01.360469  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 89.
I20250114 03:12:01.360697  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 89 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:01.360785  4567 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 89 FOLLOWER]: Leader election won for term 89
I20250114 03:12:01.360881  4567 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 89 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:01.360929  4567 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 87.90, Last appended by leader: 90, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.361409  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 88 to 89, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 89 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:01.433048  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 89 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 88. Current term is 89. Ops: []
I20250114 03:12:01.433399  4579 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250114 03:12:01.433514  4579 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 89.91 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 90
I20250114 03:12:01.433550  4579 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 88 LEADER]: Stepping down as leader of term 88
I20250114 03:12:01.433579  4579 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 88 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:01.433629  4579 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 88.91, Last appended by leader: 91, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.433698  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 88 FOLLOWER]: Advancing to term 89
I20250114 03:12:01.459812  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 89 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 88 index: 91. Preceding OpId from leader: term: 89 index: 91. (term mismatch)
I20250114 03:12:01.459892  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 90
I20250114 03:12:01.459905  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 88.91
I20250114 03:12:01.459923  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 89 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:01.460150  4567 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250114 03:12:01.461131  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 89 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 87 index: 90. Preceding OpId from leader: term: 89 index: 91. (index mismatch)
I20250114 03:12:01.461361  4584 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250114 03:12:01.653898  4583 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 89 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:01.654002  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 89 FOLLOWER]: Advancing to term 90
I20250114 03:12:01.654778  4583 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.654956  4583 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 90 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:01.655207  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 90 candidate_status { last_received { term: 89 index: 91 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:01.655215  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 90 candidate_status { last_received { term: 89 index: 91 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:01.655493  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 90 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:01.655611  4583 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 90 FOLLOWER]: Leader election lost for term 90. Reason: could not achieve majority
I20250114 03:12:01.675618  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 90 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 89. Current term is 90. Ops: []
I20250114 03:12:01.675884  4584 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 89.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250114 03:12:01.676000  4584 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 89 LEADER]: Stepping down as leader of term 89
I20250114 03:12:01.676017  4584 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 89 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:01.676074  4584 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 89.91, Last appended by leader: 91, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.676162  4584 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 89 FOLLOWER]: Advancing to term 90
I20250114 03:12:01.735530  4579 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 89 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:01.735634  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 89 FOLLOWER]: Advancing to term 90
I20250114 03:12:01.736291  4579 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.736456  4579 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 90 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:01.736680  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 90 candidate_status { last_received { term: 89 index: 91 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:01.736677  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 90 candidate_status { last_received { term: 89 index: 91 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:01.736792  4424 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 90 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 90: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:01.737365  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 90.
I20250114 03:12:01.737542  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 90 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:01.737634  4579 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 90 FOLLOWER]: Leader election won for term 90
I20250114 03:12:01.737699  4579 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 90 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:01.737754  4579 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 89.91, Last appended by leader: 91, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.738281  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 89 to 90, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 90 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:01.767442  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:01.799273  4583 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 90 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:01.799387  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 90 FOLLOWER]: Advancing to term 91
I20250114 03:12:01.800089  4583 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.800275  4583 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 91 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:01.800527  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 91 candidate_status { last_received { term: 89 index: 91 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:01.800510  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 91 candidate_status { last_received { term: 89 index: 91 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:01.800632  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 90 FOLLOWER]: Advancing to term 91
I20250114 03:12:01.801182  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 91 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 91.
I20250114 03:12:01.801366  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 91 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:01.801453  4583 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 91 FOLLOWER]: Leader election won for term 91
I20250114 03:12:01.801508  4583 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 91 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:01.801550  4583 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 89.91, Last appended by leader: 91, Current term: 91, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.802062  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 90 to 91, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 91 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:01.823873  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 91 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 90. Current term is 91. Ops: []
I20250114 03:12:01.824187  4579 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250114 03:12:01.824345  4579 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 90 LEADER]: Stepping down as leader of term 90
I20250114 03:12:01.824378  4579 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 90 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:01.824450  4579 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 90.92, Last appended by leader: 92, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:01.824542  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 90 FOLLOWER]: Advancing to term 91
I20250114 03:12:01.912513  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 91 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 89 index: 91. Preceding OpId from leader: term: 91 index: 92. (index mismatch)
I20250114 03:12:01.912827  4583 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250114 03:12:01.914192  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 91 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 90 index: 92. Preceding OpId from leader: term: 91 index: 92. (term mismatch)
I20250114 03:12:01.914253  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 91
I20250114 03:12:01.914275  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 90.92
I20250114 03:12:01.914305  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 91 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:01.914578  4585 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250114 03:12:02.099325  4579 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 91 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:02.099431  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 91 FOLLOWER]: Advancing to term 92
I20250114 03:12:02.100123  4579 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.100278  4579 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 92 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:02.100443  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 92 candidate_status { last_received { term: 91 index: 92 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:02.100447  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 92 candidate_status { last_received { term: 91 index: 92 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:02.100742  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 92 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:02.100832  4579 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 92 FOLLOWER]: Leader election lost for term 92. Reason: could not achieve majority
I20250114 03:12:02.111259  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 92 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 91. Current term is 92. Ops: []
I20250114 03:12:02.111524  4583 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 91.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250114 03:12:02.111639  4583 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 91 LEADER]: Stepping down as leader of term 91
I20250114 03:12:02.111670  4583 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 91 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:02.111742  4583 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 91.92, Last appended by leader: 92, Current term: 91, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.111816  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 91 FOLLOWER]: Advancing to term 92
I20250114 03:12:02.199870  4584 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 91 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:02.199972  4584 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 91 FOLLOWER]: Advancing to term 92
I20250114 03:12:02.200667  4584 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.200837  4584 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 92 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:02.201074  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 92 candidate_status { last_received { term: 91 index: 92 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:02.201074  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 92 candidate_status { last_received { term: 91 index: 92 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:02.201215  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 92: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:02.201761  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 92 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 92.
I20250114 03:12:02.201959  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 92 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:02.202051  4584 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 92 FOLLOWER]: Leader election won for term 92
I20250114 03:12:02.202154  4584 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 92 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:02.202194  4584 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 91.92, Last appended by leader: 92, Current term: 92, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.202715  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 91 to 92, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 92 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:02.218585  4579 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 92 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:02.218698  4579 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 92 FOLLOWER]: Advancing to term 93
I20250114 03:12:02.219388  4579 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.219563  4579 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 93 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:02.219774  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 93 candidate_status { last_received { term: 91 index: 92 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:02.219796  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 93 candidate_status { last_received { term: 91 index: 92 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:02.219874  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 92 FOLLOWER]: Advancing to term 93
I20250114 03:12:02.220423  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 93.
I20250114 03:12:02.220589  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 93 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:02.220677  4579 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 93 FOLLOWER]: Leader election won for term 93
I20250114 03:12:02.220731  4579 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 93 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:02.220793  4579 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 91.92, Last appended by leader: 92, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.221313  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 92 to 93, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 93 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:02.288676  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 93 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 92. Current term is 93. Ops: []
I20250114 03:12:02.288961  4584 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250114 03:12:02.289083  4584 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 92 LEADER]: Stepping down as leader of term 92
I20250114 03:12:02.289101  4584 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 92 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:02.289155  4584 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 92.93, Last appended by leader: 93, Current term: 92, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.289247  4584 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 92 FOLLOWER]: Advancing to term 93
I20250114 03:12:02.316637  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 93 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 91 index: 92. Preceding OpId from leader: term: 93 index: 93. (index mismatch)
I20250114 03:12:02.316956  4579 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250114 03:12:02.317925  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 93 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 92 index: 93. Preceding OpId from leader: term: 93 index: 93. (term mismatch)
I20250114 03:12:02.318078  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 92
I20250114 03:12:02.318140  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 92.93
I20250114 03:12:02.318174  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 93 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:02.318387  4579 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250114 03:12:02.528532  4584 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 93 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:02.528635  4584 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 93 FOLLOWER]: Advancing to term 94
I20250114 03:12:02.529310  4584 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.529503  4584 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 94 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:02.529732  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 94 candidate_status { last_received { term: 93 index: 93 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:02.529732  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 94 candidate_status { last_received { term: 93 index: 93 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:02.529986  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:02.530068  4584 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20250114 03:12:02.537514  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 94 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 93. Current term is 94. Ops: []
I20250114 03:12:02.537885  4587 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 93.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250114 03:12:02.538012  4587 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 93 LEADER]: Stepping down as leader of term 93
I20250114 03:12:02.538111  4587 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 93 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:02.538208  4587 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 93.93, Last appended by leader: 93, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.538301  4587 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 93 FOLLOWER]: Advancing to term 94
I20250114 03:12:02.588565  4583 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 93 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:02.588673  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 93 FOLLOWER]: Advancing to term 94
I20250114 03:12:02.589370  4583 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.589529  4583 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 94 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:02.589808  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 94 candidate_status { last_received { term: 93 index: 93 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:02.589808  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 94 candidate_status { last_received { term: 93 index: 93 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:02.589983  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 94: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:02.590605  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 94 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 94.
I20250114 03:12:02.590783  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 94 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:02.590891  4583 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 94 FOLLOWER]: Leader election won for term 94
I20250114 03:12:02.590966  4583 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 94 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:02.591012  4583 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 93.93, Last appended by leader: 93, Current term: 94, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.591519  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 93 to 94, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 94 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:02.655596  4584 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 94 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:02.655699  4584 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 94 FOLLOWER]: Advancing to term 95
I20250114 03:12:02.656380  4584 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.656543  4584 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 95 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:02.656802  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 95 candidate_status { last_received { term: 93 index: 93 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:02.656800  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 95 candidate_status { last_received { term: 93 index: 93 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:02.656917  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 94 FOLLOWER]: Advancing to term 95
I20250114 03:12:02.657490  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 95.
I20250114 03:12:02.657677  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 95 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:02.657764  4584 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 95 FOLLOWER]: Leader election won for term 95
I20250114 03:12:02.657814  4584 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 95 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:02.657898  4584 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 93.93, Last appended by leader: 93, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.658569  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 94 to 95, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 95 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:02.685015  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 95 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 94. Current term is 95. Ops: []
I20250114 03:12:02.685401  4583 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250114 03:12:02.685523  4583 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 95.94 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 93
I20250114 03:12:02.685565  4583 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 94 LEADER]: Stepping down as leader of term 94
I20250114 03:12:02.685614  4583 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 94 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:02.685670  4583 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 94.94, Last appended by leader: 94, Current term: 94, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.685750  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 94 FOLLOWER]: Advancing to term 95
I20250114 03:12:02.742192  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 95 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 94 index: 94. Preceding OpId from leader: term: 95 index: 94. (term mismatch)
I20250114 03:12:02.742273  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 93
I20250114 03:12:02.742295  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 94.94
I20250114 03:12:02.742318  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 95 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:02.742547  4584 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250114 03:12:02.743813  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 95 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 93 index: 93. Preceding OpId from leader: term: 95 index: 94. (index mismatch)
I20250114 03:12:02.744004  4588 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250114 03:12:02.767732  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:02.943743  4582 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 95 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:02.943847  4582 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 95 FOLLOWER]: Advancing to term 96
I20250114 03:12:02.944590  4582 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.944746  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 96 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 95. Current term is 96. Ops: []
I20250114 03:12:02.944751  4582 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 96 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:02.944907  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 96 candidate_status { last_received { term: 95 index: 94 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:02.944907  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 96 candidate_status { last_received { term: 95 index: 94 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:02.945055  4586 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 95.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250114 03:12:02.945148  4588 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 95 LEADER]: Stepping down as leader of term 95
I20250114 03:12:02.945217  4588 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 95 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:02.945207  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 96 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:02.945305  4582 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 96 FOLLOWER]: Leader election lost for term 96. Reason: could not achieve majority
I20250114 03:12:02.945323  4588 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 95.94, Last appended by leader: 94, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:02.945418  4588 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 95 FOLLOWER]: Advancing to term 96
I20250114 03:12:03.021649  4583 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 95 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:03.021752  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 95 FOLLOWER]: Advancing to term 96
I20250114 03:12:03.022457  4583 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.022641  4583 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 96 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:03.022850  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 96 candidate_status { last_received { term: 95 index: 94 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:03.022919  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 96 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 96: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:03.022876  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 96 candidate_status { last_received { term: 95 index: 94 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:03.023483  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 96.
I20250114 03:12:03.023665  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 96 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:03.023753  4583 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 96 FOLLOWER]: Leader election won for term 96
I20250114 03:12:03.023818  4583 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 96 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:03.023877  4583 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 95.94, Last appended by leader: 94, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.024499  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 95 to 96, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 96 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:03.069293  4582 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 96 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:03.069406  4582 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 96 FOLLOWER]: Advancing to term 97
I20250114 03:12:03.070163  4582 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.070335  4582 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 97 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:03.070560  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 97 candidate_status { last_received { term: 95 index: 94 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:03.070562  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 97 candidate_status { last_received { term: 95 index: 94 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:03.070658  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 96 FOLLOWER]: Advancing to term 97
I20250114 03:12:03.071208  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 97 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 97.
I20250114 03:12:03.071370  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 97 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:03.071456  4582 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 97 FOLLOWER]: Leader election won for term 97
I20250114 03:12:03.071511  4582 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 97 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:03.071563  4582 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 95.94, Last appended by leader: 94, Current term: 97, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.072032  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 96 to 97, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 97 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:03.122785  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 97 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 96. Current term is 97. Ops: []
I20250114 03:12:03.123066  4583 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250114 03:12:03.123184  4583 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 97.95 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 94
I20250114 03:12:03.123227  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 96 LEADER]: Stepping down as leader of term 96
I20250114 03:12:03.123270  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 96 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:03.123332  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 96.95, Last appended by leader: 95, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.123426  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 96 FOLLOWER]: Advancing to term 97
I20250114 03:12:03.192780  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 97 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 95 index: 94. Preceding OpId from leader: term: 97 index: 95. (index mismatch)
I20250114 03:12:03.193109  4582 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250114 03:12:03.194229  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 97 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 96 index: 95. Preceding OpId from leader: term: 97 index: 95. (term mismatch)
I20250114 03:12:03.194293  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 94
I20250114 03:12:03.194319  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 96.95
I20250114 03:12:03.194337  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 97 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:03.194499  4582 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250114 03:12:03.475818  4588 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 97 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:03.475907  4588 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 97 FOLLOWER]: Advancing to term 98
I20250114 03:12:03.476588  4588 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.476759  4588 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 98 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:03.476953  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 98 candidate_status { last_received { term: 97 index: 95 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:03.476958  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 98 candidate_status { last_received { term: 97 index: 95 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:03.477252  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 98 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:03.477336  4588 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 98 FOLLOWER]: Leader election lost for term 98. Reason: could not achieve majority
I20250114 03:12:03.481520  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 98 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 97. Current term is 98. Ops: []
I20250114 03:12:03.481786  4589 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 97.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250114 03:12:03.481932  4589 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 97 LEADER]: Stepping down as leader of term 97
I20250114 03:12:03.481971  4589 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 97 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:03.482024  4589 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 97.95, Last appended by leader: 95, Current term: 97, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.482112  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 97 FOLLOWER]: Advancing to term 98
I20250114 03:12:03.575451  4583 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 97 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:03.575557  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 97 FOLLOWER]: Advancing to term 98
I20250114 03:12:03.576200  4583 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.576347  4583 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 98 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:03.576582  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 98 candidate_status { last_received { term: 97 index: 95 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:03.576603  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 98 candidate_status { last_received { term: 97 index: 95 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:03.576679  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 98 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 98: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:03.577203  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 98.
I20250114 03:12:03.577383  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 98 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:03.577462  4583 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 98 FOLLOWER]: Leader election won for term 98
I20250114 03:12:03.577523  4583 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 98 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:03.577580  4583 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 97.95, Last appended by leader: 95, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.578070  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 97 to 98, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 98 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:03.614001  4588 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 98 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:03.614115  4588 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 98 FOLLOWER]: Advancing to term 99
I20250114 03:12:03.614831  4588 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.615044  4588 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 99 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:03.615239  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 99 candidate_status { last_received { term: 97 index: 95 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:03.615239  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 99 candidate_status { last_received { term: 97 index: 95 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:03.615422  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 98 FOLLOWER]: Advancing to term 99
I20250114 03:12:03.615957  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 99 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 99.
I20250114 03:12:03.616144  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 99 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:03.616240  4588 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 99 FOLLOWER]: Leader election won for term 99
I20250114 03:12:03.616357  4588 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 99 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:03.616418  4588 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 97.95, Last appended by leader: 95, Current term: 99, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.616825  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 98 to 99, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 99 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:03.665563  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 99 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 98. Current term is 99. Ops: []
I20250114 03:12:03.665886  4583 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250114 03:12:03.666014  4583 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 98 LEADER]: Stepping down as leader of term 98
I20250114 03:12:03.666034  4583 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 98 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:03.666083  4583 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 98.96, Last appended by leader: 96, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.666162  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 98 FOLLOWER]: Advancing to term 99
I20250114 03:12:03.722992  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 99 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 98 index: 96. Preceding OpId from leader: term: 99 index: 96. (term mismatch)
I20250114 03:12:03.723073  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 95
I20250114 03:12:03.723086  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 98.96
I20250114 03:12:03.723104  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 99 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:03.723398  4588 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250114 03:12:03.724574  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 99 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 97 index: 95. Preceding OpId from leader: term: 99 index: 96. (index mismatch)
I20250114 03:12:03.724858  4593 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250114 03:12:03.768026  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:03.935808  4583 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 99 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:03.935922  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 99 FOLLOWER]: Advancing to term 100
I20250114 03:12:03.936591  4583 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.936758  4583 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 100 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:03.936997  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 100 candidate_status { last_received { term: 99 index: 96 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:03.936997  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 100 candidate_status { last_received { term: 99 index: 96 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:03.937237  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 100 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:03.937359  4583 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 100 FOLLOWER]: Leader election lost for term 100. Reason: could not achieve majority
I20250114 03:12:03.945940  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 100 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 99. Current term is 100. Ops: []
I20250114 03:12:03.946211  4588 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 99.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250114 03:12:03.946326  4588 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 99 LEADER]: Stepping down as leader of term 99
I20250114 03:12:03.946360  4588 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 99 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:03.946424  4588 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 99.96, Last appended by leader: 96, Current term: 99, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.946502  4588 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 99 FOLLOWER]: Advancing to term 100
I20250114 03:12:03.980422  4589 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 99 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:03.980527  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 99 FOLLOWER]: Advancing to term 100
I20250114 03:12:03.981261  4589 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.981442  4589 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 100 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:03.981716  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 100 candidate_status { last_received { term: 99 index: 96 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:03.981716  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 100 candidate_status { last_received { term: 99 index: 96 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:03.981905  4424 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 100 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 100: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:03.982514  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 100.
I20250114 03:12:03.982699  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 100 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:03.982798  4589 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 100 FOLLOWER]: Leader election won for term 100
I20250114 03:12:03.982897  4589 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 100 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:03.982946  4589 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 99.96, Last appended by leader: 96, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:03.983505  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 99 to 100, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 100 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:04.074432  4583 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 100 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:04.074543  4583 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 100 FOLLOWER]: Advancing to term 101
I20250114 03:12:04.075264  4583 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.075557  4583 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 101 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:04.075649  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 101 candidate_status { last_received { term: 99 index: 96 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:04.075784  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 101 candidate_status { last_received { term: 99 index: 96 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:04.075886  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 100 FOLLOWER]: Advancing to term 101
I20250114 03:12:04.076428  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 101 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 101.
I20250114 03:12:04.076627  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 101 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:04.076716  4583 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 101 FOLLOWER]: Leader election won for term 101
I20250114 03:12:04.076774  4583 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 101 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:04.076813  4583 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 99.96, Last appended by leader: 96, Current term: 101, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.077358  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 100 to 101, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 101 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:04.083875  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 101 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 100. Current term is 101. Ops: []
I20250114 03:12:04.084183  4589 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250114 03:12:04.084286  4589 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 100 LEADER]: Stepping down as leader of term 100
I20250114 03:12:04.084306  4589 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 100 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:04.084372  4589 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 100.97, Last appended by leader: 97, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.084444  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 100 FOLLOWER]: Advancing to term 101
I20250114 03:12:04.173842  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 101 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 100 index: 97. Preceding OpId from leader: term: 101 index: 97. (term mismatch)
I20250114 03:12:04.173946  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 96
I20250114 03:12:04.173960  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 100.97
I20250114 03:12:04.173976  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 101 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:04.174315  4583 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250114 03:12:04.175534  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 101 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 99 index: 96. Preceding OpId from leader: term: 101 index: 97. (index mismatch)
I20250114 03:12:04.175815  4583 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250114 03:12:04.276424  4594 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 101 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:04.276548  4594 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 101 FOLLOWER]: Advancing to term 102
I20250114 03:12:04.277310  4594 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.277498  4594 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 102 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:04.277741  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 102 candidate_status { last_received { term: 101 index: 97 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:04.277741  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 102 candidate_status { last_received { term: 101 index: 97 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:04.277913  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 101 FOLLOWER]: Advancing to term 102
I20250114 03:12:04.278564  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 102.
I20250114 03:12:04.278764  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 102 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:04.278859  4594 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 102 FOLLOWER]: Leader election won for term 102
I20250114 03:12:04.278911  4594 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 102 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:04.278956  4594 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 101.97, Last appended by leader: 97, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.279510  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 101 to 102, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 102 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:04.285548  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 102 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 101. Current term is 102. Ops: []
I20250114 03:12:04.285809  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 101.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250114 03:12:04.285943  4561 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 102.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
I20250114 03:12:04.285988  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 101 LEADER]: Stepping down as leader of term 101
I20250114 03:12:04.286020  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 101 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:04.286082  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 101.97, Last appended by leader: 97, Current term: 101, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.286167  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 101 FOLLOWER]: Advancing to term 102
I20250114 03:12:04.376586  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 102 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 101 index: 97. Preceding OpId from leader: term: 102 index: 98. (index mismatch)
I20250114 03:12:04.376852  4594 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250114 03:12:04.377954  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 102 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 101 index: 97. Preceding OpId from leader: term: 102 index: 98. (index mismatch)
I20250114 03:12:04.378268  4594 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250114 03:12:04.479286  4595 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 102 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:04.479393  4595 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 102 FOLLOWER]: Advancing to term 103
I20250114 03:12:04.480213  4595 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.480381  4595 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 103 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:04.480583  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:04.480592  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:04.480836  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 103 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:04.480933  4595 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 103 FOLLOWER]: Leader election lost for term 103. Reason: could not achieve majority
I20250114 03:12:04.493757  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 103 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 102. Current term is 103. Ops: []
I20250114 03:12:04.494122  4597 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 102.98, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250114 03:12:04.494257  4597 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 102 LEADER]: Stepping down as leader of term 102
I20250114 03:12:04.494289  4597 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 102 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:04.494372  4597 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 102.98, Last appended by leader: 98, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.494585  4597 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 102 FOLLOWER]: Advancing to term 103
I20250114 03:12:04.564589  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 102 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:04.564693  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 102 FOLLOWER]: Advancing to term 103
I20250114 03:12:04.565332  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.565485  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 103 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:04.565721  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:04.565797  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 103: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:04.565728  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 103 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:04.566426  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 103.
I20250114 03:12:04.566617  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 103 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:04.566715  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 103 FOLLOWER]: Leader election won for term 103
I20250114 03:12:04.566777  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 103 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:04.566818  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 102.98, Last appended by leader: 98, Current term: 103, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.567274  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 102 to 103, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 103 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:04.629331  4595 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:04.629413  4595 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 103 FOLLOWER]: Advancing to term 104
I20250114 03:12:04.630103  4595 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.630264  4595 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 104 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:04.630496  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 104 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:04.630496  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 104 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:04.630614  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 103 FOLLOWER]: Advancing to term 104
I20250114 03:12:04.631155  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 104.
I20250114 03:12:04.631317  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:04.631449  4595 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 104 FOLLOWER]: Leader election won for term 104
I20250114 03:12:04.631544  4595 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 104 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:04.631601  4595 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 102.98, Last appended by leader: 98, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.632076  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 103 to 104, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 104 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:04.644634  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 104 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 103. Current term is 104. Ops: []
I20250114 03:12:04.644894  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250114 03:12:04.644985  4561 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 104.99 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 98
I20250114 03:12:04.645027  4596 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 103 LEADER]: Stepping down as leader of term 103
I20250114 03:12:04.645072  4596 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 103 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:04.645136  4596 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 103.99, Last appended by leader: 99, Current term: 103, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.645215  4596 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 103 FOLLOWER]: Advancing to term 104
I20250114 03:12:04.737187  4597 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 104 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:04.737303  4597 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 104 FOLLOWER]: Advancing to term 105
I20250114 03:12:04.738130  4597 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.738312  4597 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 105 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:04.738557  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 105 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:04.738626  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 104 FOLLOWER]: Advancing to term 105
I20250114 03:12:04.738590  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 105 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:04.739315  4424 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 for term 105 because replica has last-logged OpId of term: 103 index: 99, which is greater than that of the candidate, which has last-logged OpId of term: 102 index: 98.
I20250114 03:12:04.739506  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:04.739603  4597 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20250114 03:12:04.742672  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 105 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 104. Current term is 105. Ops: []
I20250114 03:12:04.742972  4595 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250114 03:12:04.743062  4595 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 103.99 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 98
I20250114 03:12:04.743112  4589 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 104 LEADER]: Stepping down as leader of term 104
I20250114 03:12:04.743152  4589 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:04.743204  4589 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 104.99, Last appended by leader: 99, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.743281  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 104 FOLLOWER]: Advancing to term 105
I20250114 03:12:04.768337  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:04.889104  4597 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:04.889217  4597 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 105 FOLLOWER]: Advancing to term 106
I20250114 03:12:04.889964  4597 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.890131  4597 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 106 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:04.890385  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 106 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:04.890388  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 106 candidate_status { last_received { term: 102 index: 98 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:04.890497  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 105 FOLLOWER]: Advancing to term 106
I20250114 03:12:04.890506  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 105 FOLLOWER]: Advancing to term 106
I20250114 03:12:04.891103  4424 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 for term 106 because replica has last-logged OpId of term: 103 index: 99, which is greater than that of the candidate, which has last-logged OpId of term: 102 index: 98.
I20250114 03:12:04.891186  4273 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 for term 106 because replica has last-logged OpId of term: 104 index: 99, which is greater than that of the candidate, which has last-logged OpId of term: 102 index: 98.
I20250114 03:12:04.891371  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 106 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:04.891469  4597 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 106 FOLLOWER]: Leader election lost for term 106. Reason: could not achieve majority
I20250114 03:12:04.972525  4596 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 106 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:04.972635  4596 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 106 FOLLOWER]: Advancing to term 107
I20250114 03:12:04.973518  4596 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.973719  4596 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 107 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:04.973978  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 107 candidate_status { last_received { term: 103 index: 99 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:04.974073  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 106 FOLLOWER]: Advancing to term 107
I20250114 03:12:04.974033  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 107 candidate_status { last_received { term: 103 index: 99 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:04.974128  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 106 FOLLOWER]: Advancing to term 107
I20250114 03:12:04.974678  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 107.
I20250114 03:12:04.974737  4273 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a for term 107 because replica has last-logged OpId of term: 104 index: 99, which is greater than that of the candidate, which has last-logged OpId of term: 103 index: 99.
I20250114 03:12:04.974865  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 107 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 
I20250114 03:12:04.974965  4596 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 107 FOLLOWER]: Leader election won for term 107
I20250114 03:12:04.975031  4596 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 107 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:04.975066  4596 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 103.99, Last appended by leader: 99, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.975698  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 104 to 107, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 107 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:04.987006  4589 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 107 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:04.987102  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 107 FOLLOWER]: Advancing to term 108
I20250114 03:12:04.987785  4589 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.987943  4589 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 108 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:04.988161  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 108 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:04.988224  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 107 FOLLOWER]: Advancing to term 108
I20250114 03:12:04.988166  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 108 candidate_status { last_received { term: 104 index: 99 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:04.988760  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 108 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 108.
I20250114 03:12:04.988935  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 108 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:04.989030  4589 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 108 FOLLOWER]: Leader election won for term 108
I20250114 03:12:04.989085  4589 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 108 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:04.989125  4589 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 104.99, Last appended by leader: 99, Current term: 108, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:04.989621  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 107 to 108, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 108 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:05.068084  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 108 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 107. Current term is 108. Ops: []
I20250114 03:12:05.068387  4596 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 100, Last known committed idx: 98, Time since last communication: 0.000s
I20250114 03:12:05.068506  4596 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 107 LEADER]: Stepping down as leader of term 107
I20250114 03:12:05.068524  4596 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 107 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:05.068586  4596 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 107.100, Last appended by leader: 100, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.068660  4596 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 107 FOLLOWER]: Advancing to term 108
I20250114 03:12:05.094293  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 108 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 102 index: 98. Preceding OpId from leader: term: 108 index: 100. (index mismatch)
I20250114 03:12:05.094606  4589 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 98, Time since last communication: 0.000s
I20250114 03:12:05.095664  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 108 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 107 index: 100. Preceding OpId from leader: term: 108 index: 100. (term mismatch)
I20250114 03:12:05.095724  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 99
I20250114 03:12:05.095746  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 107.100
I20250114 03:12:05.095769  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:05.095927  4595 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 98, Time since last communication: 0.000s
I20250114 03:12:05.095995  4595 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 103.99 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 98
I20250114 03:12:05.096163  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 98
I20250114 03:12:05.096208  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 103.99
I20250114 03:12:05.096223  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:05.363329  4599 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 108 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:05.363415  4599 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 108 FOLLOWER]: Advancing to term 109
I20250114 03:12:05.364172  4599 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.364344  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 109 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 108. Current term is 109. Ops: []
I20250114 03:12:05.364339  4599 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 109 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:05.364600  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:05.364575  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:05.364727  4589 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 108.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250114 03:12:05.364861  4598 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 108 LEADER]: Stepping down as leader of term 108
I20250114 03:12:05.364912  4598 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 108 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:05.364917  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 109 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:05.365017  4599 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 109 FOLLOWER]: Leader election lost for term 109. Reason: could not achieve majority
I20250114 03:12:05.364996  4598 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 108, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.365119  4598 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 108 FOLLOWER]: Advancing to term 109
I20250114 03:12:05.401584  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 108 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:05.401703  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 108 FOLLOWER]: Advancing to term 109
I20250114 03:12:05.402477  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.402664  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 109 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:05.402884  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:05.402884  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:05.403043  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 109 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 109: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:05.403555  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 109.
I20250114 03:12:05.403720  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:05.403856  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 109 FOLLOWER]: Leader election won for term 109
I20250114 03:12:05.403928  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 109 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:05.403990  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.404449  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 108 to 109, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 109 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:05.470444  4599 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 109 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:05.470564  4599 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 109 FOLLOWER]: Advancing to term 110
I20250114 03:12:05.471215  4599 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.471402  4599 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 110 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:05.471644  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 110 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:05.471714  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 109 FOLLOWER]: Advancing to term 110
I20250114 03:12:05.471649  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 110 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:05.472322  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 110 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 110.
I20250114 03:12:05.472501  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 110 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:05.472622  4599 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 110 FOLLOWER]: Leader election won for term 110
I20250114 03:12:05.472698  4599 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 110 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:05.472765  4599 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 110, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.473241  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 109 to 110, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 110 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:05.496995  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 110 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 109. Current term is 110. Ops: []
I20250114 03:12:05.497320  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250114 03:12:05.497431  4561 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 110.101 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 100
I20250114 03:12:05.497475  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 109 LEADER]: Stepping down as leader of term 109
I20250114 03:12:05.497501  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 109 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:05.497556  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 109.101, Last appended by leader: 101, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.497630  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 109 FOLLOWER]: Advancing to term 110
I20250114 03:12:05.567869  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 110 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 108 index: 100. Preceding OpId from leader: term: 110 index: 101. (index mismatch)
I20250114 03:12:05.568192  4599 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250114 03:12:05.569398  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 110 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 109 index: 101. Preceding OpId from leader: term: 110 index: 101. (term mismatch)
I20250114 03:12:05.569459  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 100
I20250114 03:12:05.569479  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 109.101
I20250114 03:12:05.569497  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 110 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:05.569717  4597 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250114 03:12:05.669800  4598 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 110 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:05.669920  4598 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 110 FOLLOWER]: Advancing to term 111
I20250114 03:12:05.670600  4598 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.670871  4598 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 111 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:05.671042  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 111 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:05.671002  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 111 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:05.671290  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 111 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:05.671418  4598 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 111 FOLLOWER]: Leader election lost for term 111. Reason: could not achieve majority
I20250114 03:12:05.686676  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 111 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 110. Current term is 111. Ops: []
I20250114 03:12:05.686954  4599 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 110.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250114 03:12:05.687115  4600 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 110 LEADER]: Stepping down as leader of term 110
I20250114 03:12:05.687170  4600 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 110 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:05.687242  4600 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 110.101, Last appended by leader: 101, Current term: 110, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.687333  4600 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 110 FOLLOWER]: Advancing to term 111
I20250114 03:12:05.761206  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 110 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:05.761305  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 110 FOLLOWER]: Advancing to term 111
I20250114 03:12:05.762039  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.762195  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 111 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:05.762423  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 111 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:05.762428  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 111 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:05.762586  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 111 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 111: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:05.763100  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 111.
I20250114 03:12:05.763260  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:05.763347  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 111 FOLLOWER]: Leader election won for term 111
I20250114 03:12:05.763401  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 111 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:05.763445  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 110.101, Last appended by leader: 101, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.763938  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 110 to 111, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 111 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:05.768591  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:05.781999  4598 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 111 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:05.782095  4598 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 111 FOLLOWER]: Advancing to term 112
I20250114 03:12:05.782835  4598 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.783004  4598 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 112 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:05.783221  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 112 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:05.783286  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 111 FOLLOWER]: Advancing to term 112
I20250114 03:12:05.783226  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 112 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:05.783851  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 112 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 112.
I20250114 03:12:05.784029  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 112 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:05.784129  4598 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 112 FOLLOWER]: Leader election won for term 112
I20250114 03:12:05.784188  4598 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 112 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:05.784236  4598 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 110.101, Last appended by leader: 101, Current term: 112, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.784720  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 111 to 112, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 112 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:05.860250  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 112 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 111. Current term is 112. Ops: []
I20250114 03:12:05.860538  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250114 03:12:05.860669  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 111 LEADER]: Stepping down as leader of term 111
I20250114 03:12:05.860708  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 111 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:05.860759  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 111.102, Last appended by leader: 102, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.860817  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 111 FOLLOWER]: Advancing to term 112
I20250114 03:12:05.874053  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 112 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 111 index: 102. Preceding OpId from leader: term: 112 index: 102. (term mismatch)
I20250114 03:12:05.874127  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 101
I20250114 03:12:05.874141  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 111.102
I20250114 03:12:05.874157  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 112 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:05.874387  4598 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250114 03:12:05.875455  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 112 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 110 index: 101. Preceding OpId from leader: term: 112 index: 102. (index mismatch)
I20250114 03:12:05.875655  4601 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250114 03:12:05.976294  4600 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 112 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:05.976405  4600 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 112 FOLLOWER]: Advancing to term 113
I20250114 03:12:05.977102  4600 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.977289  4600 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 113 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:05.977523  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 113 candidate_status { last_received { term: 112 index: 102 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:05.977523  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 113 candidate_status { last_received { term: 112 index: 102 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:05.977656  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 112 FOLLOWER]: Advancing to term 113
I20250114 03:12:05.978276  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 113.
I20250114 03:12:05.978453  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:05.978587  4600 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 113 FOLLOWER]: Leader election won for term 113
I20250114 03:12:05.978665  4600 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 113 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:05.978725  4600 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 112.102, Last appended by leader: 102, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.979223  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 112 to 113, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 113 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:05.988243  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 113 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 112. Current term is 113. Ops: []
I20250114 03:12:05.988247  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 113 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 112. Current term is 113. Ops: []
I20250114 03:12:05.988473  4601 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 112.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250114 03:12:05.988575  4601 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 113.103 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 102
I20250114 03:12:05.988610  4589 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 112 LEADER]: Stepping down as leader of term 112
I20250114 03:12:05.988667  4589 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 112 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:05.988615  4598 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 112.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250114 03:12:05.988780  4589 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 112.102, Last appended by leader: 102, Current term: 112, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:05.988875  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 112 FOLLOWER]: Advancing to term 113
I20250114 03:12:06.054565  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 113 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 112 index: 102. Preceding OpId from leader: term: 113 index: 103. (index mismatch)
I20250114 03:12:06.054836  4600 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250114 03:12:06.055934  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 113 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 112 index: 102. Preceding OpId from leader: term: 113 index: 103. (index mismatch)
I20250114 03:12:06.056180  4600 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250114 03:12:06.156622  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 113 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:06.156738  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 113 FOLLOWER]: Advancing to term 114
I20250114 03:12:06.157425  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.157616  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 114 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:06.157842  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:06.157843  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:06.157977  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 113 FOLLOWER]: Advancing to term 114
I20250114 03:12:06.158555  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 114 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 114.
I20250114 03:12:06.158732  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 114 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:06.158835  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 114 FOLLOWER]: Leader election won for term 114
I20250114 03:12:06.158888  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 114 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:06.158941  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.103, Last appended by leader: 103, Current term: 114, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.159435  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 113 to 114, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 114 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:06.167644  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 114 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 113. Current term is 114. Ops: []
I20250114 03:12:06.167972  4600 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 113.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250114 03:12:06.168083  4600 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 114.104 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 103
I20250114 03:12:06.168118  4599 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 113 LEADER]: Stepping down as leader of term 113
I20250114 03:12:06.168169  4599 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 113 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:06.168238  4599 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.103, Last appended by leader: 103, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.168339  4599 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 113 FOLLOWER]: Advancing to term 114
I20250114 03:12:06.240605  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 114 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 113 index: 103. Preceding OpId from leader: term: 114 index: 104. (index mismatch)
I20250114 03:12:06.240900  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250114 03:12:06.241797  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 114 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 113 index: 103. Preceding OpId from leader: term: 114 index: 104. (index mismatch)
I20250114 03:12:06.242045  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250114 03:12:06.342676  4601 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 114 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:06.342795  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 114 FOLLOWER]: Advancing to term 115
I20250114 03:12:06.343508  4601 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.343798  4601 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 115 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:06.343990  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:06.343990  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:06.344259  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 115 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:06.344359  4601 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 115 FOLLOWER]: Leader election lost for term 115. Reason: could not achieve majority
I20250114 03:12:06.358104  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 115 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 114. Current term is 115. Ops: []
I20250114 03:12:06.358371  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 114.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250114 03:12:06.358502  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 114 LEADER]: Stepping down as leader of term 114
I20250114 03:12:06.358546  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 114 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:06.358608  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 114.104, Last appended by leader: 104, Current term: 114, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.358688  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 114 FOLLOWER]: Advancing to term 115
I20250114 03:12:06.426100  4600 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 114 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:06.426199  4600 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 114 FOLLOWER]: Advancing to term 115
I20250114 03:12:06.426887  4600 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.427062  4600 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 115 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:06.427270  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:06.427347  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 115: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:06.427275  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 115 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:06.427937  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 115.
I20250114 03:12:06.428117  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:06.428202  4600 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 115 FOLLOWER]: Leader election won for term 115
I20250114 03:12:06.428252  4600 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 115 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:06.428293  4600 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 114.104, Last appended by leader: 104, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.428761  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 114 to 115, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 115 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:06.449821  4601 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 115 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:06.449950  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 115 FOLLOWER]: Advancing to term 116
I20250114 03:12:06.450644  4601 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.450791  4601 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 116 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:06.451047  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 116 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:06.451047  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 116 candidate_status { last_received { term: 114 index: 104 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:06.451171  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 115 FOLLOWER]: Advancing to term 116
I20250114 03:12:06.451720  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 116 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 116.
I20250114 03:12:06.451915  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 116 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:06.452003  4601 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 116 FOLLOWER]: Leader election won for term 116
I20250114 03:12:06.452055  4601 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 116 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:06.452111  4601 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 114.104, Last appended by leader: 104, Current term: 116, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.452594  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 115 to 116, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 116 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:06.505903  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 116 LEADER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 115. Current term is 116. Ops: []
I20250114 03:12:06.506207  4600 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250114 03:12:06.506332  4600 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 116.105 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 104
I20250114 03:12:06.506376  4599 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 115 LEADER]: Stepping down as leader of term 115
I20250114 03:12:06.506430  4599 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 115 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:06.506500  4599 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 115.105, Last appended by leader: 105, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.506606  4599 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 115 FOLLOWER]: Advancing to term 116
I20250114 03:12:06.529032  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 116 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 115 index: 105. Preceding OpId from leader: term: 116 index: 105. (term mismatch)
I20250114 03:12:06.529114  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 104
I20250114 03:12:06.529129  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 115.105
I20250114 03:12:06.529145  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 116 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:06.529377  4601 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250114 03:12:06.530406  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 116 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 114 index: 104. Preceding OpId from leader: term: 116 index: 105. (index mismatch)
I20250114 03:12:06.530709  4602 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250114 03:12:06.631767  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 116 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:06.631860  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 116 FOLLOWER]: Advancing to term 117
I20250114 03:12:06.632591  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.632776  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 117 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:06.633011  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:06.633030  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:06.633270  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 117 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:06.633369  4561 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 117 FOLLOWER]: Leader election lost for term 117. Reason: could not achieve majority
I20250114 03:12:06.634693  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 117 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 116. Current term is 117. Ops: []
I20250114 03:12:06.634938  4589 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 116.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250114 03:12:06.635030  4589 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 116 LEADER]: Stepping down as leader of term 116
I20250114 03:12:06.635056  4589 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 116 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:06.635103  4589 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 116.105, Last appended by leader: 105, Current term: 116, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.635195  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 116 FOLLOWER]: Advancing to term 117
I20250114 03:12:06.732570  4599 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 116 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:06.732688  4599 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 116 FOLLOWER]: Advancing to term 117
I20250114 03:12:06.733386  4599 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.733577  4599 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 117 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:06.733805  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:06.733816  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 117 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:06.733937  4424 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 117 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 117: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:06.734691  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 117.
I20250114 03:12:06.734902  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:06.734997  4599 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 117 FOLLOWER]: Leader election won for term 117
I20250114 03:12:06.735062  4599 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 117 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:06.735123  4599 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 116.105, Last appended by leader: 105, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.735601  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 116 to 117, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 117 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:06.738724  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 117 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:06.738786  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 117 FOLLOWER]: Advancing to term 118
I20250114 03:12:06.739434  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.739589  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 118 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:06.739743  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 118 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:06.739756  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 118 candidate_status { last_received { term: 116 index: 105 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:06.739840  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 117 FOLLOWER]: Advancing to term 118
I20250114 03:12:06.740460  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 118 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 118.
I20250114 03:12:06.740725  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 118 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:06.740854  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 118 FOLLOWER]: Leader election won for term 118
I20250114 03:12:06.740908  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 118 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:06.740967  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 116.105, Last appended by leader: 105, Current term: 118, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.741411  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 117 to 118, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 118 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:06.768821  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:06.817718  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 118 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 116 index: 105. Preceding OpId from leader: term: 118 index: 106. (index mismatch)
I20250114 03:12:06.818044  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250114 03:12:06.819283  4347 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 117 LEADER]: Stepping down as leader of term 117
I20250114 03:12:06.819348  4347 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 117 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:06.819399  4347 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 117.106, Last appended by leader: 106, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.819525  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 117 FOLLOWER]: Advancing to term 118
I20250114 03:12:06.820148  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 118 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 117 index: 106. Preceding OpId from leader: term: 118 index: 106. (term mismatch)
I20250114 03:12:06.820175  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 105
I20250114 03:12:06.820187  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 117.106
I20250114 03:12:06.820201  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 118 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:06.820505  4603 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250114 03:12:06.919999  4589 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 118 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:06.920094  4589 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 118 FOLLOWER]: Advancing to term 119
I20250114 03:12:06.921439  4589 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.921622  4589 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 119 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:06.921847  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 119 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:06.921880  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 119 candidate_status { last_received { term: 118 index: 106 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:06.921985  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 118 FOLLOWER]: Advancing to term 119
I20250114 03:12:06.922586  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 119.
I20250114 03:12:06.922762  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 119 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:06.922855  4589 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 119 FOLLOWER]: Leader election won for term 119
I20250114 03:12:06.922911  4589 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 119 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:06.922968  4589 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 118.106, Last appended by leader: 106, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.923472  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 118 to 119, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 119 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:06.927171  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 119 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 118. Current term is 119. Ops: []
I20250114 03:12:06.927404  4603 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 118.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250114 03:12:06.927496  4603 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 5eea63c188224be98e6888084ea42808 log is divergent from this leader: its last log entry 119.107 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 106
I20250114 03:12:06.927538  4561 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 118 LEADER]: Stepping down as leader of term 118
I20250114 03:12:06.927577  4561 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 118 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:06.927634  4561 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 118.106, Last appended by leader: 106, Current term: 118, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:06.927716  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 118 FOLLOWER]: Advancing to term 119
I20250114 03:12:07.022109  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 119 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 118 index: 106. Preceding OpId from leader: term: 119 index: 107. (index mismatch)
I20250114 03:12:07.022432  4589 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250114 03:12:07.023514  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 119 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 118 index: 106. Preceding OpId from leader: term: 119 index: 107. (index mismatch)
I20250114 03:12:07.023728  4602 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250114 03:12:07.201460  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 119 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:07.201572  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 119 FOLLOWER]: Advancing to term 120
I20250114 03:12:07.202282  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.202440  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 120 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:07.202669  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 120 candidate_status { last_received { term: 119 index: 107 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:07.202672  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 120 candidate_status { last_received { term: 119 index: 107 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:07.202930  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 120 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:07.203032  4561 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 120 FOLLOWER]: Leader election lost for term 120. Reason: could not achieve majority
I20250114 03:12:07.211711  4599 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 119 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:07.211792  4599 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 119 FOLLOWER]: Advancing to term 120
I20250114 03:12:07.212427  4599 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.212626  4599 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 120 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:07.212829  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 120 candidate_status { last_received { term: 119 index: 107 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:07.212829  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 120 candidate_status { last_received { term: 119 index: 107 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:07.212953  4424 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 120 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 120: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:07.213102  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 120 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:07.213225  4599 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 120 FOLLOWER]: Leader election lost for term 120. Reason: could not achieve majority
I20250114 03:12:07.222206  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 120 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 119. Current term is 120. Ops: []
I20250114 03:12:07.222450  4601 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 119.107, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250114 03:12:07.222585  4601 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 119 LEADER]: Stepping down as leader of term 119
I20250114 03:12:07.222625  4601 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 119 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:07.222683  4601 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 119.107, Last appended by leader: 107, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.222774  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 119 FOLLOWER]: Advancing to term 120
I20250114 03:12:07.345679  4599 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 120 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:07.345782  4599 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 120 FOLLOWER]: Advancing to term 121
I20250114 03:12:07.346503  4599 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.346668  4599 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 121 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:07.346901  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 121 candidate_status { last_received { term: 119 index: 107 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:07.346971  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 120 FOLLOWER]: Advancing to term 121
I20250114 03:12:07.346920  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 121 candidate_status { last_received { term: 119 index: 107 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:07.347014  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 120 FOLLOWER]: Advancing to term 121
I20250114 03:12:07.347536  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 121.
I20250114 03:12:07.347568  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 121.
I20250114 03:12:07.347735  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 121 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 
I20250114 03:12:07.347831  4599 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 121 FOLLOWER]: Leader election won for term 121
I20250114 03:12:07.347931  4599 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 121 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:07.347975  4599 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 119.107, Last appended by leader: 107, Current term: 121, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.348409  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 119 to 121, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 121 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:07.430943  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 121 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 119 index: 107. Preceding OpId from leader: term: 121 index: 108. (index mismatch)
I20250114 03:12:07.431216  4599 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250114 03:12:07.432327  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 121 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 119 index: 107. Preceding OpId from leader: term: 121 index: 108. (index mismatch)
I20250114 03:12:07.432545  4605 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250114 03:12:07.532895  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 121 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:07.532989  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 121 FOLLOWER]: Advancing to term 122
I20250114 03:12:07.533672  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.533839  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 122 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:07.534049  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 122 candidate_status { last_received { term: 121 index: 108 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:07.534049  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 122 candidate_status { last_received { term: 121 index: 108 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:07.534332  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 122 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:07.534443  4561 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 122 FOLLOWER]: Leader election lost for term 122. Reason: could not achieve majority
I20250114 03:12:07.544093  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 122 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 121. Current term is 122. Ops: []
I20250114 03:12:07.544328  4604 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 121.108, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250114 03:12:07.544431  4604 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 121 LEADER]: Stepping down as leader of term 121
I20250114 03:12:07.544468  4604 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 121 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:07.544529  4604 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 121.108, Last appended by leader: 108, Current term: 121, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.544606  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 121 FOLLOWER]: Advancing to term 122
I20250114 03:12:07.627756  4601 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 121 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:07.627861  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 121 FOLLOWER]: Advancing to term 122
I20250114 03:12:07.628553  4601 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.628726  4601 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 122 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:07.628973  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 122 candidate_status { last_received { term: 121 index: 108 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:07.628978  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 122 candidate_status { last_received { term: 121 index: 108 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:07.629052  4424 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 122 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 122: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:07.629657  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 122.
I20250114 03:12:07.629844  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:07.629963  4601 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 122 FOLLOWER]: Leader election won for term 122
I20250114 03:12:07.630017  4601 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 122 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:07.630059  4601 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 121.108, Last appended by leader: 108, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.630618  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 121 to 122, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 122 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:07.680054  4561 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 122 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:07.680162  4561 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 122 FOLLOWER]: Advancing to term 123
I20250114 03:12:07.680907  4561 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.681077  4561 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 123 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:07.681290  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 123 candidate_status { last_received { term: 121 index: 108 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:07.681313  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 123 candidate_status { last_received { term: 121 index: 108 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:07.681422  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 122 FOLLOWER]: Advancing to term 123
I20250114 03:12:07.682087  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 123 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 123.
I20250114 03:12:07.682246  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 123 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:07.682335  4561 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 123 FOLLOWER]: Leader election won for term 123
I20250114 03:12:07.682435  4561 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 123 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:07.682494  4561 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 121.108, Last appended by leader: 108, Current term: 123, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.682911  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 122 to 123, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 123 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:07.714027  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 123 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 122. Current term is 123. Ops: []
I20250114 03:12:07.714295  4601 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250114 03:12:07.714402  4601 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 123.109 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 108
I20250114 03:12:07.714432  4602 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 122 LEADER]: Stepping down as leader of term 122
I20250114 03:12:07.714480  4602 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 122 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:07.714561  4602 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 122.109, Last appended by leader: 109, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.714658  4602 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 122 FOLLOWER]: Advancing to term 123
I20250114 03:12:07.769102  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:07.774536  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 123 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 121 index: 108. Preceding OpId from leader: term: 123 index: 109. (index mismatch)
I20250114 03:12:07.774839  4561 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250114 03:12:07.776031  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 123 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 122 index: 109. Preceding OpId from leader: term: 123 index: 109. (term mismatch)
I20250114 03:12:07.776103  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 108
I20250114 03:12:07.776124  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 122.109
I20250114 03:12:07.776145  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 123 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:07.776305  4607 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250114 03:12:07.876564  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 123 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:07.876657  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 123 FOLLOWER]: Advancing to term 124
I20250114 03:12:07.877306  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.877456  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 124 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:07.877681  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 124 candidate_status { last_received { term: 123 index: 109 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:07.877681  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 124 candidate_status { last_received { term: 123 index: 109 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:07.878046  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 124 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:07.878136  4604 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 124 FOLLOWER]: Leader election lost for term 124. Reason: could not achieve majority
I20250114 03:12:07.884200  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 124 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 123. Current term is 124. Ops: []
I20250114 03:12:07.884459  4561 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 123.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250114 03:12:07.884584  4606 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 123 LEADER]: Stepping down as leader of term 123
I20250114 03:12:07.884637  4606 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 123 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:07.884703  4606 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 123.109, Last appended by leader: 109, Current term: 123, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.884809  4606 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 123 FOLLOWER]: Advancing to term 124
I20250114 03:12:07.971576  4602 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 123 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:07.971696  4602 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 123 FOLLOWER]: Advancing to term 124
I20250114 03:12:07.972373  4602 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.972527  4602 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 124 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:07.972764  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 124 candidate_status { last_received { term: 123 index: 109 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:07.972848  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 124: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:07.972775  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 124 candidate_status { last_received { term: 123 index: 109 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:07.973438  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 124.
I20250114 03:12:07.973657  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 124 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:07.973757  4602 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 124 FOLLOWER]: Leader election won for term 124
I20250114 03:12:07.973830  4602 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 124 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:07.973920  4602 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 123.109, Last appended by leader: 109, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:07.974395  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 123 to 124, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 124 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:08.025753  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 124 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:08.025890  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 124 FOLLOWER]: Advancing to term 125
I20250114 03:12:08.026588  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.026751  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 125 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:08.026991  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 125 candidate_status { last_received { term: 123 index: 109 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:08.027004  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 125 candidate_status { last_received { term: 123 index: 109 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:08.027102  4424 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 124 FOLLOWER]: Advancing to term 125
I20250114 03:12:08.027685  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 125.
I20250114 03:12:08.027854  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 125 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:08.027942  4604 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 125 FOLLOWER]: Leader election won for term 125
I20250114 03:12:08.028005  4604 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 125 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:08.028065  4604 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 123.109, Last appended by leader: 109, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.028499  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 124 to 125, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 125 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:08.068825  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 125 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 124. Current term is 125. Ops: []
I20250114 03:12:08.069133  4602 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250114 03:12:08.069244  4602 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 125.110 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 109
I20250114 03:12:08.069293  4601 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 124 LEADER]: Stepping down as leader of term 124
I20250114 03:12:08.069334  4601 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 124 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:08.069386  4601 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 124.110, Last appended by leader: 110, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.069466  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 124 FOLLOWER]: Advancing to term 125
I20250114 03:12:08.120206  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 125 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 123 index: 109. Preceding OpId from leader: term: 125 index: 110. (index mismatch)
I20250114 03:12:08.120481  4604 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250114 03:12:08.121536  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 125 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 124 index: 110. Preceding OpId from leader: term: 125 index: 110. (term mismatch)
I20250114 03:12:08.121593  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 109
I20250114 03:12:08.121618  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 124.110
I20250114 03:12:08.121639  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 125 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:08.121822  4608 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250114 03:12:08.308563  4606 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 125 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:08.308686  4606 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 125 FOLLOWER]: Advancing to term 126
I20250114 03:12:08.309417  4606 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.309615  4606 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 126 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:08.309836  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 126 candidate_status { last_received { term: 125 index: 110 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:08.309834  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 126 candidate_status { last_received { term: 125 index: 110 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:08.310129  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 126 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:08.310231  4606 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 126 FOLLOWER]: Leader election lost for term 126. Reason: could not achieve majority
I20250114 03:12:08.312490  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 126 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 125. Current term is 126. Ops: []
I20250114 03:12:08.312750  4605 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 125.110, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250114 03:12:08.312873  4604 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 125 LEADER]: Stepping down as leader of term 125
I20250114 03:12:08.312920  4604 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 125 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:08.312983  4604 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 125.110, Last appended by leader: 110, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.313057  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 125 FOLLOWER]: Advancing to term 126
I20250114 03:12:08.330338  4601 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 125 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:08.330438  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 125 FOLLOWER]: Advancing to term 126
I20250114 03:12:08.331107  4601 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.331288  4601 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 126 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:08.331496  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 126 candidate_status { last_received { term: 125 index: 110 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:08.331501  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 126 candidate_status { last_received { term: 125 index: 110 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:08.331591  4424 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 126 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 126: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:08.332124  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 126.
I20250114 03:12:08.332309  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:08.332403  4601 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 126 FOLLOWER]: Leader election won for term 126
I20250114 03:12:08.332458  4601 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 126 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:08.332515  4601 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 125.110, Last appended by leader: 110, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.332996  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 125 to 126, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 126 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:08.420421  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 126 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 125 index: 110. Preceding OpId from leader: term: 126 index: 111. (index mismatch)
I20250114 03:12:08.420694  4601 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250114 03:12:08.421940  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 126 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 125 index: 110. Preceding OpId from leader: term: 126 index: 111. (index mismatch)
I20250114 03:12:08.422171  4609 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250114 03:12:08.617983  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 126 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:08.618086  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 126 FOLLOWER]: Advancing to term 127
I20250114 03:12:08.618844  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.619038  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 127 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:08.619273  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 127 candidate_status { last_received { term: 126 index: 111 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:08.619283  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 127 candidate_status { last_received { term: 126 index: 111 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:08.619498  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 127 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:08.619601  4604 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 127 FOLLOWER]: Leader election lost for term 127. Reason: could not achieve majority
I20250114 03:12:08.623015  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 127 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 126. Current term is 127. Ops: []
I20250114 03:12:08.623234  4601 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 126.111, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250114 03:12:08.623335  4601 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 126 LEADER]: Stepping down as leader of term 126
I20250114 03:12:08.623378  4601 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 126 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:08.623448  4601 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 126.111, Last appended by leader: 111, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.623520  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 126 FOLLOWER]: Advancing to term 127
I20250114 03:12:08.679745  4610 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 126 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:08.679853  4610 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 126 FOLLOWER]: Advancing to term 127
I20250114 03:12:08.680701  4610 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.680900  4610 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 127 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:08.681128  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 127 candidate_status { last_received { term: 126 index: 111 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:08.681193  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 127 candidate_status { last_received { term: 126 index: 111 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:08.681303  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 127 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 127: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:08.681902  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 127.
I20250114 03:12:08.682086  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 127 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:08.682178  4610 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 127 FOLLOWER]: Leader election won for term 127
I20250114 03:12:08.682258  4610 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 127 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:08.682304  4610 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 126.111, Last appended by leader: 111, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.682828  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 126 to 127, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 127 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:08.720018  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 127 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:08.720120  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 127 FOLLOWER]: Advancing to term 128
I20250114 03:12:08.720808  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.720978  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 128 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:08.721174  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 128 candidate_status { last_received { term: 126 index: 111 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:08.721182  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 128 candidate_status { last_received { term: 126 index: 111 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:08.721314  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 127 FOLLOWER]: Advancing to term 128
I20250114 03:12:08.721909  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 128 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 128.
I20250114 03:12:08.722055  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 128 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:08.722146  4604 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 128 FOLLOWER]: Leader election won for term 128
I20250114 03:12:08.722203  4604 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 128 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:08.722259  4604 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 126.111, Last appended by leader: 111, Current term: 128, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.722720  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 127 to 128, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 128 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:08.769222  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 128 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 127. Current term is 128. Ops: []
I20250114 03:12:08.769340  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:08.769520  4610 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250114 03:12:08.769618  4610 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 127 LEADER]: Stepping down as leader of term 127
I20250114 03:12:08.769639  4610 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 127 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:08.769691  4610 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 127.112, Last appended by leader: 112, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:08.769764  4610 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 127 FOLLOWER]: Advancing to term 128
I20250114 03:12:08.836246  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 128 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 126 index: 111. Preceding OpId from leader: term: 128 index: 112. (index mismatch)
I20250114 03:12:08.836570  4604 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250114 03:12:08.837780  4424 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 128 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 127 index: 112. Preceding OpId from leader: term: 128 index: 112. (term mismatch)
I20250114 03:12:08.837831  4424 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 111
I20250114 03:12:08.837845  4424 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 127.112
I20250114 03:12:08.837886  4424 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 128 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:08.838063  4611 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250114 03:12:09.055193  4610 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 128 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:09.055299  4610 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 128 FOLLOWER]: Advancing to term 129
I20250114 03:12:09.056001  4610 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.056160  4610 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 129 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:09.056397  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 129 candidate_status { last_received { term: 128 index: 112 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:09.056409  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 129 candidate_status { last_received { term: 128 index: 112 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:09.056697  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 129 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:09.056802  4610 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 129 FOLLOWER]: Leader election lost for term 129. Reason: could not achieve majority
I20250114 03:12:09.072278  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 129 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 128. Current term is 129. Ops: []
I20250114 03:12:09.072557  4604 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 128.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250114 03:12:09.072688  4604 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 128 LEADER]: Stepping down as leader of term 128
I20250114 03:12:09.072724  4604 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 128 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:09.072793  4604 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 128.112, Last appended by leader: 112, Current term: 128, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.072906  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 128 FOLLOWER]: Advancing to term 129
I20250114 03:12:09.121291  4601 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 128 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:09.121397  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 128 FOLLOWER]: Advancing to term 129
I20250114 03:12:09.122136  4601 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.122311  4601 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 129 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:09.122526  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 129 candidate_status { last_received { term: 128 index: 112 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:09.122542  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 129 candidate_status { last_received { term: 128 index: 112 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:09.122627  4424 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 129: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:09.123158  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 129.
I20250114 03:12:09.123317  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:09.123420  4601 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 129 FOLLOWER]: Leader election won for term 129
I20250114 03:12:09.123536  4601 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 129 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:09.123615  4601 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 128.112, Last appended by leader: 112, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.124094  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 128 to 129, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 129 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:09.176282  4610 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 129 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:09.176383  4610 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 129 FOLLOWER]: Advancing to term 130
I20250114 03:12:09.177079  4610 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.177248  4610 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 130 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:09.177528  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 130 candidate_status { last_received { term: 128 index: 112 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:09.177537  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 130 candidate_status { last_received { term: 128 index: 112 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:09.177645  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 129 FOLLOWER]: Advancing to term 130
I20250114 03:12:09.178231  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 130.
I20250114 03:12:09.178383  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 130 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:09.178468  4610 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 130 FOLLOWER]: Leader election won for term 130
I20250114 03:12:09.178534  4610 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 130 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:09.178591  4610 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 128.112, Last appended by leader: 112, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.179051  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 129 to 130, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 130 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:09.236732  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 130 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 129. Current term is 130. Ops: []
I20250114 03:12:09.237006  4601 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250114 03:12:09.237131  4601 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 129 LEADER]: Stepping down as leader of term 129
I20250114 03:12:09.237165  4424 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 130 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 129. Current term is 130. Ops: []
I20250114 03:12:09.237166  4601 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 129 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:09.237280  4601 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 129.113, Last appended by leader: 113, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.237372  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 129 FOLLOWER]: Advancing to term 130
I20250114 03:12:09.283030  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 130 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 128 index: 112. Preceding OpId from leader: term: 130 index: 113. (index mismatch)
I20250114 03:12:09.283337  4610 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250114 03:12:09.284471  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 130 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 129 index: 113. Preceding OpId from leader: term: 130 index: 113. (term mismatch)
I20250114 03:12:09.284521  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 112
I20250114 03:12:09.284570  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 129.113
I20250114 03:12:09.284593  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 130 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:09.284754  4613 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250114 03:12:09.478686  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 130 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:09.478789  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 130 FOLLOWER]: Advancing to term 131
I20250114 03:12:09.479462  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.479658  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 131 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:09.479928  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 131 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:09.479913  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 131 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:09.480177  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 131 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:09.480274  4604 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 131 FOLLOWER]: Leader election lost for term 131. Reason: could not achieve majority
I20250114 03:12:09.493155  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 131 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 130. Current term is 131. Ops: []
I20250114 03:12:09.493430  4610 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 130.113, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250114 03:12:09.493541  4610 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 130 LEADER]: Stepping down as leader of term 130
I20250114 03:12:09.493578  4610 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 130 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:09.493636  4610 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 130.113, Last appended by leader: 113, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.493719  4610 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 130 FOLLOWER]: Advancing to term 131
I20250114 03:12:09.511022  4601 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 130 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:09.511122  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 130 FOLLOWER]: Advancing to term 131
I20250114 03:12:09.511770  4601 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.511948  4601 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 131 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:09.512167  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 131 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:09.512176  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 131 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:09.512312  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 131 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 131: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:09.512868  4424 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 131.
I20250114 03:12:09.513036  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 131 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:09.513119  4601 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 131 FOLLOWER]: Leader election won for term 131
I20250114 03:12:09.513180  4601 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 131 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:09.513239  4601 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 130.113, Last appended by leader: 113, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.513736  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 130 to 131, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 131 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:09.614900  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 131 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:09.615003  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 131 FOLLOWER]: Advancing to term 132
I20250114 03:12:09.615657  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.615841  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 132 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:09.616025  4424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 132 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:09.616106  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 131 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 130 index: 113. Preceding OpId from leader: term: 131 index: 114. (index mismatch)
I20250114 03:12:09.616166  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 132 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:09.616318  4601 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250114 03:12:09.616364  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 132 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:09.616465  4604 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 132 FOLLOWER]: Leader election lost for term 132. Reason: could not achieve majority
I20250114 03:12:09.617240  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 132 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 131. Current term is 132. Ops: []
I20250114 03:12:09.617477  4612 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250114 03:12:09.617601  4601 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 131 LEADER]: Stepping down as leader of term 131
I20250114 03:12:09.617647  4601 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 131 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:09.617705  4601 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 131.114, Last appended by leader: 114, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.617779  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 131 FOLLOWER]: Advancing to term 132
I20250114 03:12:09.719070  4610 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 131 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:09.719173  4610 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 131 FOLLOWER]: Advancing to term 132
I20250114 03:12:09.719833  4610 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.719986  4610 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 132 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:09.720196  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 132 candidate_status { last_received { term: 131 index: 114 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:09.720208  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 132 candidate_status { last_received { term: 131 index: 114 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:09.720296  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 132 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 132: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:09.720876  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 132.
I20250114 03:12:09.721041  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:09.721129  4610 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 132 FOLLOWER]: Leader election won for term 132
I20250114 03:12:09.721200  4610 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 132 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:09.721256  4610 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 131.114, Last appended by leader: 114, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.721730  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 131 to 132, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 132 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:09.725598  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 132 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:09.725662  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 132 FOLLOWER]: Advancing to term 133
I20250114 03:12:09.726239  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.726378  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 133 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:09.726564  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 133 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:09.726581  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 133 candidate_status { last_received { term: 130 index: 113 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:09.726683  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 132 FOLLOWER]: Advancing to term 133
I20250114 03:12:09.727201  4273 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 for term 133 because replica has last-logged OpId of term: 131 index: 114, which is greater than that of the candidate, which has last-logged OpId of term: 130 index: 113.
I20250114 03:12:09.727377  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:09.727469  4604 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I20250114 03:12:09.769629  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:09.806313  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 133 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 132. Current term is 133. Ops: []
I20250114 03:12:09.806629  4610 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.000s
I20250114 03:12:09.806820  4606 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 132 LEADER]: Stepping down as leader of term 132
I20250114 03:12:09.806874  4606 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 132 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:09.806932  4606 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 132.115, Last appended by leader: 115, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.807025  4606 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 132 FOLLOWER]: Advancing to term 133
I20250114 03:12:09.843364  4601 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 133 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:09.843463  4601 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 133 FOLLOWER]: Advancing to term 134
I20250114 03:12:09.844141  4601 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.844312  4601 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 134 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:09.844552  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 134 candidate_status { last_received { term: 131 index: 114 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:09.844620  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 133 FOLLOWER]: Advancing to term 134
I20250114 03:12:09.844594  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 134 candidate_status { last_received { term: 131 index: 114 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:09.844717  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 133 FOLLOWER]: Advancing to term 134
I20250114 03:12:09.845228  4425 raft_consensus.cc:2405] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 134 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 for term 134 because replica has last-logged OpId of term: 132 index: 115, which is greater than that of the candidate, which has last-logged OpId of term: 131 index: 114.
I20250114 03:12:09.845232  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 134.
I20250114 03:12:09.845444  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 134 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:09.845539  4601 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 134 FOLLOWER]: Leader election won for term 134
I20250114 03:12:09.845602  4601 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 134 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:09.845650  4601 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 131.114, Last appended by leader: 114, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:09.846122  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 132 to 134, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 134 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:09.941601  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 134 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 132 index: 115. Preceding OpId from leader: term: 134 index: 115. (term mismatch)
I20250114 03:12:09.941680  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 114
I20250114 03:12:09.941704  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 132.115
I20250114 03:12:09.941723  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 134 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:09.942041  4601 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250114 03:12:09.943223  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 134 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 130 index: 113. Preceding OpId from leader: term: 134 index: 115. (index mismatch)
I20250114 03:12:09.943420  4612 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.000s
I20250114 03:12:10.043582  4606 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 134 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:10.043677  4606 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 134 FOLLOWER]: Advancing to term 135
I20250114 03:12:10.044360  4606 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.044520  4606 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 135 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:10.044777  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 135 candidate_status { last_received { term: 134 index: 115 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:10.044777  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 135 candidate_status { last_received { term: 134 index: 115 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:10.044871  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 134 FOLLOWER]: Advancing to term 135
I20250114 03:12:10.045431  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 135 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 135.
I20250114 03:12:10.045630  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 135 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:10.045737  4606 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 135 FOLLOWER]: Leader election won for term 135
I20250114 03:12:10.045802  4606 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 135 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:10.045883  4606 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 134.115, Last appended by leader: 115, Current term: 135, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.046314  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 134 to 135, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 135 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:10.049798  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 135 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 134. Current term is 135. Ops: []
I20250114 03:12:10.050019  4614 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 134.115, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250114 03:12:10.050130  4614 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 135.116 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 115
I20250114 03:12:10.050181  4612 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 134 LEADER]: Stepping down as leader of term 134
I20250114 03:12:10.050241  4612 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 134 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:10.050299  4612 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 134.115, Last appended by leader: 115, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.050387  4612 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 134 FOLLOWER]: Advancing to term 135
I20250114 03:12:10.127583  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 135 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 134 index: 115. Preceding OpId from leader: term: 135 index: 116. (index mismatch)
I20250114 03:12:10.127866  4606 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250114 03:12:10.128986  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 135 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 134 index: 115. Preceding OpId from leader: term: 135 index: 116. (index mismatch)
I20250114 03:12:10.129205  4606 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250114 03:12:10.229892  4615 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 135 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:10.230059  4615 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 135 FOLLOWER]: Advancing to term 136
I20250114 03:12:10.230861  4615 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.230995  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 136 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 135. Current term is 136. Ops: []
I20250114 03:12:10.231088  4615 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 136 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:10.231176  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:10.231210  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:10.231333  4616 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 135.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250114 03:12:10.231375  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 136 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:10.231423  4616 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 135 LEADER]: Stepping down as leader of term 135
I20250114 03:12:10.231464  4615 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 136 FOLLOWER]: Leader election lost for term 136. Reason: could not achieve majority
I20250114 03:12:10.231473  4616 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 135 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:10.231623  4616 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 135.116, Last appended by leader: 116, Current term: 135, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.231699  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 135 FOLLOWER]: Advancing to term 136
I20250114 03:12:10.316159  4612 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 135 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:10.316272  4612 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 135 FOLLOWER]: Advancing to term 136
I20250114 03:12:10.317052  4612 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.317210  4612 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 136 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:10.317432  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:10.317443  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:10.317556  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 136 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 136: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:10.318081  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 136.
I20250114 03:12:10.318249  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:10.318342  4612 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 136 FOLLOWER]: Leader election won for term 136
I20250114 03:12:10.318393  4612 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 136 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:10.318452  4612 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 135.116, Last appended by leader: 116, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.318909  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 135 to 136, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 136 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:10.332944  4615 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 136 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:10.333058  4615 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 136 FOLLOWER]: Advancing to term 137
I20250114 03:12:10.333803  4615 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.333995  4615 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 137 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:10.334250  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 137 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:10.334250  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 137 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:10.334369  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 136 FOLLOWER]: Advancing to term 137
I20250114 03:12:10.334962  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 137 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 137.
I20250114 03:12:10.335155  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 137 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:10.335242  4615 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 137 FOLLOWER]: Leader election won for term 137
I20250114 03:12:10.335291  4615 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 137 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:10.335361  4615 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 135.116, Last appended by leader: 116, Current term: 137, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.335811  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 136 to 137, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 137 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:10.417300  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 137 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 135 index: 116. Preceding OpId from leader: term: 137 index: 117. (index mismatch)
I20250114 03:12:10.417626  4615 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250114 03:12:10.418744  4273 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 136 LEADER]: Stepping down as leader of term 136
I20250114 03:12:10.418809  4273 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 136 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:10.418871  4273 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 136.117, Last appended by leader: 117, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.418947  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 136 FOLLOWER]: Advancing to term 137
I20250114 03:12:10.419615  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 137 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 136 index: 117. Preceding OpId from leader: term: 137 index: 117. (term mismatch)
I20250114 03:12:10.419654  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 116
I20250114 03:12:10.419677  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 136.117
I20250114 03:12:10.419696  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 137 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:10.419930  4617 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250114 03:12:10.611527  4616 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 137 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:10.611645  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 137 FOLLOWER]: Advancing to term 138
I20250114 03:12:10.612320  4616 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.612484  4616 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 138 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:10.612721  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 138 candidate_status { last_received { term: 137 index: 117 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:10.612736  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 138 candidate_status { last_received { term: 137 index: 117 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:10.612803  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 137 FOLLOWER]: Advancing to term 138
I20250114 03:12:10.613444  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 138.
I20250114 03:12:10.613598  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:10.613688  4616 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 138 FOLLOWER]: Leader election won for term 138
I20250114 03:12:10.613737  4616 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 138 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:10.613795  4616 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 137.117, Last appended by leader: 117, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.614275  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 137 to 138, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 138 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:10.616392  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 138 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 137. Current term is 138. Ops: []
I20250114 03:12:10.616586  4604 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 137.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250114 03:12:10.616677  4604 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 137 LEADER]: Stepping down as leader of term 137
I20250114 03:12:10.616708  4604 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 137 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:10.616758  4604 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 137.117, Last appended by leader: 117, Current term: 137, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.616843  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 137 FOLLOWER]: Advancing to term 138
I20250114 03:12:10.698940  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 138 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 137 index: 117. Preceding OpId from leader: term: 138 index: 118. (index mismatch)
I20250114 03:12:10.699230  4616 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250114 03:12:10.700264  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 138 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 137 index: 117. Preceding OpId from leader: term: 138 index: 118. (index mismatch)
I20250114 03:12:10.700546  4606 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250114 03:12:10.769919  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:10.801802  4614 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 138 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:10.801962  4614 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 138 FOLLOWER]: Advancing to term 139
I20250114 03:12:10.802781  4614 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.802955  4614 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 139 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:10.803160  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:10.803231  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:10.803436  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:10.803527  4614 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I20250114 03:12:10.813962  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 139 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 138. Current term is 139. Ops: []
I20250114 03:12:10.814188  4616 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 138.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250114 03:12:10.814317  4616 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 138 LEADER]: Stepping down as leader of term 138
I20250114 03:12:10.814353  4616 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 138 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:10.814410  4616 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 138.118, Last appended by leader: 118, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.814491  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 138 FOLLOWER]: Advancing to term 139
I20250114 03:12:10.887915  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 138 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:10.888017  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 138 FOLLOWER]: Advancing to term 139
I20250114 03:12:10.888736  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.888911  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 139 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:10.889168  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:10.889173  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:10.889286  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 139: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:10.889904  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 139.
I20250114 03:12:10.890079  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:10.890158  4604 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 139 FOLLOWER]: Leader election won for term 139
I20250114 03:12:10.890220  4604 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 139 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:10.890278  4604 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 138.118, Last appended by leader: 118, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.891036  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 138 to 139, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 139 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:10.945997  4614 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:10.946099  4614 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 139 FOLLOWER]: Advancing to term 140
I20250114 03:12:10.946817  4614 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.947028  4614 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 140 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:10.947279  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 140 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:10.947285  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 140 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:10.947396  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 139 FOLLOWER]: Advancing to term 140
I20250114 03:12:10.948006  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 140 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 140.
I20250114 03:12:10.948177  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 140 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:10.948259  4614 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 140 FOLLOWER]: Leader election won for term 140
I20250114 03:12:10.948381  4614 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 140 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:10.948529  4614 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 138.118, Last appended by leader: 118, Current term: 140, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.949003  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 139 to 140, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 140 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:10.992123  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 140 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 139. Current term is 140. Ops: []
I20250114 03:12:10.992405  4604 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250114 03:12:10.992552  4604 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 139 LEADER]: Stepping down as leader of term 139
I20250114 03:12:10.992590  4604 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 139 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:10.992658  4604 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 139.119, Last appended by leader: 119, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:10.992743  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 139 FOLLOWER]: Advancing to term 140
I20250114 03:12:11.031421  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 140 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 138 index: 118. Preceding OpId from leader: term: 140 index: 119. (index mismatch)
I20250114 03:12:11.031723  4614 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250114 03:12:11.032740  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 140 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 139 index: 119. Preceding OpId from leader: term: 140 index: 119. (term mismatch)
I20250114 03:12:11.032802  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 118
I20250114 03:12:11.032816  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 139.119
I20250114 03:12:11.032831  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 140 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:11.033033  4614 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250114 03:12:11.228384  4616 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 140 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:11.228494  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 140 FOLLOWER]: Advancing to term 141
I20250114 03:12:11.229249  4616 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.229444  4616 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 141 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:11.229660  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:11.229660  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:11.229950  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 141 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:11.230054  4616 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 141 FOLLOWER]: Leader election lost for term 141. Reason: could not achieve majority
I20250114 03:12:11.243045  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 141 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 140. Current term is 141. Ops: []
I20250114 03:12:11.243309  4618 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 140.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250114 03:12:11.243425  4618 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 140 LEADER]: Stepping down as leader of term 140
I20250114 03:12:11.243460  4618 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 140 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:11.243510  4618 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 140.119, Last appended by leader: 119, Current term: 140, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.243592  4618 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 140 FOLLOWER]: Advancing to term 141
I20250114 03:12:11.310140  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 140 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:11.310243  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 140 FOLLOWER]: Advancing to term 141
I20250114 03:12:11.310889  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.311041  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 141 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:11.311270  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:11.311270  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:11.311419  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 141 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 141: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:11.311977  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 141.
I20250114 03:12:11.312148  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 141 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:11.312258  4604 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 141 FOLLOWER]: Leader election won for term 141
I20250114 03:12:11.312317  4604 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 141 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:11.312369  4604 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 140.119, Last appended by leader: 119, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.312855  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 140 to 141, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 141 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:11.379055  4616 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 141 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:11.379160  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 141 FOLLOWER]: Advancing to term 142
I20250114 03:12:11.379889  4616 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.380060  4616 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 142 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:11.380268  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 142 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:11.380278  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 142 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:11.380378  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 141 FOLLOWER]: Advancing to term 142
I20250114 03:12:11.380945  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 142.
I20250114 03:12:11.381112  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:11.381186  4616 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 142 FOLLOWER]: Leader election won for term 142
I20250114 03:12:11.381314  4616 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 142 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:11.381369  4616 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 140.119, Last appended by leader: 119, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.381784  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 141 to 142, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 142 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:11.389003  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 142 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 141. Current term is 142. Ops: []
I20250114 03:12:11.389214  4604 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250114 03:12:11.389318  4604 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 141 LEADER]: Stepping down as leader of term 141
I20250114 03:12:11.389355  4604 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 141 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:11.389406  4604 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 141.120, Last appended by leader: 120, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.389480  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 141 FOLLOWER]: Advancing to term 142
I20250114 03:12:11.462280  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 142 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 140 index: 119. Preceding OpId from leader: term: 142 index: 120. (index mismatch)
I20250114 03:12:11.462563  4616 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250114 03:12:11.463649  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 142 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 141 index: 120. Preceding OpId from leader: term: 142 index: 120. (term mismatch)
I20250114 03:12:11.463698  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 119
I20250114 03:12:11.463719  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 141.120
I20250114 03:12:11.463742  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 142 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:11.463979  4621 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250114 03:12:11.754492  4618 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 142 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:11.754601  4618 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 142 FOLLOWER]: Advancing to term 143
I20250114 03:12:11.755304  4618 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.755491  4618 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 143 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:11.755707  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:11.755707  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:11.755983  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 143 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:11.756062  4618 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 143 FOLLOWER]: Leader election lost for term 143. Reason: could not achieve majority
I20250114 03:12:11.758708  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 143 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 142. Current term is 143. Ops: []
I20250114 03:12:11.758901  4620 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 142.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250114 03:12:11.759016  4620 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 142 LEADER]: Stepping down as leader of term 142
I20250114 03:12:11.759081  4620 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 142 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:11.759152  4620 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 142.120, Last appended by leader: 120, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.759228  4620 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 142 FOLLOWER]: Advancing to term 143
I20250114 03:12:11.766090  4604 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 142 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:11.766162  4604 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 142 FOLLOWER]: Advancing to term 143
I20250114 03:12:11.766752  4604 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.766899  4604 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 143 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:11.767062  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:11.767062  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:11.767163  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 143 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 143: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:11.767635  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 143 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 143.
I20250114 03:12:11.767820  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 143 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:11.767916  4604 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 143 FOLLOWER]: Leader election won for term 143
I20250114 03:12:11.767982  4604 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 143 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:11.768038  4604 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 142.120, Last appended by leader: 120, Current term: 143, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:11.768481  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 142 to 143, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 143 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:11.770186  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:11.851976  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 143 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 142 index: 120. Preceding OpId from leader: term: 143 index: 121. (index mismatch)
I20250114 03:12:11.852244  4604 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250114 03:12:11.853387  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 143 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 142 index: 120. Preceding OpId from leader: term: 143 index: 121. (index mismatch)
I20250114 03:12:11.853662  4604 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250114 03:12:12.032100  4618 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 143 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:12.032188  4618 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 143 FOLLOWER]: Advancing to term 144
I20250114 03:12:12.032859  4618 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.033020  4618 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 144 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:12.033206  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 144 candidate_status { last_received { term: 143 index: 121 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:12.033213  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 144 candidate_status { last_received { term: 143 index: 121 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:12.033427  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 144 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:12.033525  4618 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 144 FOLLOWER]: Leader election lost for term 144. Reason: could not achieve majority
I20250114 03:12:12.048691  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 144 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 143. Current term is 144. Ops: []
I20250114 03:12:12.048961  4622 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 143.121, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250114 03:12:12.049085  4622 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 143 LEADER]: Stepping down as leader of term 143
I20250114 03:12:12.049124  4622 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 143 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:12.049180  4622 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 143.121, Last appended by leader: 121, Current term: 143, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.049264  4622 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 143 FOLLOWER]: Advancing to term 144
I20250114 03:12:12.130975  4620 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 143 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:12.131088  4620 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 143 FOLLOWER]: Advancing to term 144
I20250114 03:12:12.131752  4620 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.132011  4620 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 144 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:12.132225  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 144 candidate_status { last_received { term: 143 index: 121 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:12.132251  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 144 candidate_status { last_received { term: 143 index: 121 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:12.132323  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 144 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 144: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:12.132884  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 144.
I20250114 03:12:12.133061  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 144 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:12.133149  4620 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 144 FOLLOWER]: Leader election won for term 144
I20250114 03:12:12.133203  4620 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 144 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:12.133244  4620 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 143.121, Last appended by leader: 121, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.133698  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 143 to 144, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 144 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:12.165036  4618 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 144 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:12.165138  4618 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 144 FOLLOWER]: Advancing to term 145
I20250114 03:12:12.165926  4618 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.166167  4618 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 145 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:12.166353  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 145 candidate_status { last_received { term: 143 index: 121 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:12.166353  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 145 candidate_status { last_received { term: 143 index: 121 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:12.166452  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 144 FOLLOWER]: Advancing to term 145
I20250114 03:12:12.167044  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 145 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 145.
I20250114 03:12:12.167338  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 145 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:12.167467  4618 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 145 FOLLOWER]: Leader election won for term 145
I20250114 03:12:12.167590  4618 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 145 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:12.167654  4618 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 143.121, Last appended by leader: 121, Current term: 145, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.168118  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 144 to 145, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 145 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:12.210170  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 145 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 144. Current term is 145. Ops: []
I20250114 03:12:12.210438  4620 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250114 03:12:12.210565  4616 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 144 LEADER]: Stepping down as leader of term 144
I20250114 03:12:12.210608  4616 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 144 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:12.210667  4616 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 144.122, Last appended by leader: 122, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.210749  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 144 FOLLOWER]: Advancing to term 145
I20250114 03:12:12.245604  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 145 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 144 index: 122. Preceding OpId from leader: term: 145 index: 122. (term mismatch)
I20250114 03:12:12.245692  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 121
I20250114 03:12:12.245707  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 144.122
I20250114 03:12:12.245728  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 145 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:12.245985  4618 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250114 03:12:12.247180  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 145 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 143 index: 121. Preceding OpId from leader: term: 145 index: 122. (index mismatch)
I20250114 03:12:12.247512  4624 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250114 03:12:12.440586  4622 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 145 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:12.440673  4622 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 145 FOLLOWER]: Advancing to term 146
I20250114 03:12:12.441349  4622 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.441519  4622 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 146 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:12.441761  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 146 candidate_status { last_received { term: 145 index: 122 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:12.441783  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 146 candidate_status { last_received { term: 145 index: 122 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:12.442034  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 146 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:12.442128  4622 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 146 FOLLOWER]: Leader election lost for term 146. Reason: could not achieve majority
I20250114 03:12:12.443274  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 146 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 145. Current term is 146. Ops: []
I20250114 03:12:12.443467  4618 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 145.122, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250114 03:12:12.443583  4624 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 145 LEADER]: Stepping down as leader of term 145
I20250114 03:12:12.443625  4624 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 145 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:12.443678  4624 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 145.122, Last appended by leader: 122, Current term: 145, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.443754  4624 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 145 FOLLOWER]: Advancing to term 146
I20250114 03:12:12.525434  4616 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 145 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:12.525537  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 145 FOLLOWER]: Advancing to term 146
I20250114 03:12:12.526228  4616 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.526377  4616 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 146 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:12.526600  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 146 candidate_status { last_received { term: 145 index: 122 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:12.526600  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 146 candidate_status { last_received { term: 145 index: 122 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:12.526716  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 146 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 146: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:12.527240  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 146.
I20250114 03:12:12.527410  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 146 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:12.527490  4616 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 146 FOLLOWER]: Leader election won for term 146
I20250114 03:12:12.527554  4616 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 146 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:12.527594  4616 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 145.122, Last appended by leader: 122, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.528038  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 145 to 146, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 146 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:12.574527  4622 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 146 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:12.574633  4622 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 146 FOLLOWER]: Advancing to term 147
I20250114 03:12:12.575300  4622 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.575495  4622 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 147 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:12.575726  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 147 candidate_status { last_received { term: 145 index: 122 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:12.575796  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 146 FOLLOWER]: Advancing to term 147
I20250114 03:12:12.575729  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 147 candidate_status { last_received { term: 145 index: 122 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:12.576344  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 147.
I20250114 03:12:12.576493  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:12.576581  4622 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 147 FOLLOWER]: Leader election won for term 147
I20250114 03:12:12.576712  4622 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 147 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:12.576787  4622 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 145.122, Last appended by leader: 122, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.577239  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 146 to 147, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 147 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:12.606523  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 147 LEADER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 146. Current term is 147. Ops: []
I20250114 03:12:12.606822  4616 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250114 03:12:12.606948  4616 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 147.123 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 122
I20250114 03:12:12.607007  4620 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 146 LEADER]: Stepping down as leader of term 146
I20250114 03:12:12.607055  4620 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 146 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:12.607111  4620 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 146.123, Last appended by leader: 123, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.607194  4620 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 146 FOLLOWER]: Advancing to term 147
I20250114 03:12:12.657737  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 147 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 145 index: 122. Preceding OpId from leader: term: 147 index: 123. (index mismatch)
I20250114 03:12:12.658073  4622 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250114 03:12:12.659214  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 147 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 146 index: 123. Preceding OpId from leader: term: 147 index: 123. (term mismatch)
I20250114 03:12:12.659390  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 122
I20250114 03:12:12.659418  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 146.123
I20250114 03:12:12.659446  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 147 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:12.659657  4622 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250114 03:12:12.770404  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:12.942090  4616 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 147 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:12.942198  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 147 FOLLOWER]: Advancing to term 148
I20250114 03:12:12.942911  4616 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.943069  4616 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 148 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:12.943276  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 148 candidate_status { last_received { term: 147 index: 123 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:12.943277  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 148 candidate_status { last_received { term: 147 index: 123 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:12.943612  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 148 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:12.943706  4616 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 148 FOLLOWER]: Leader election lost for term 148. Reason: could not achieve majority
I20250114 03:12:12.961745  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 148 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 147. Current term is 148. Ops: []
I20250114 03:12:12.962069  4625 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 147.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250114 03:12:12.962234  4626 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 147 LEADER]: Stepping down as leader of term 147
I20250114 03:12:12.962282  4626 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 147 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:12.962347  4626 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 147.123, Last appended by leader: 123, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:12.962446  4626 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 147 FOLLOWER]: Advancing to term 148
I20250114 03:12:13.031904  4624 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 147 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:13.032019  4624 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 147 FOLLOWER]: Advancing to term 148
I20250114 03:12:13.032713  4624 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.032891  4624 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 148 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:13.033118  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 148 candidate_status { last_received { term: 147 index: 123 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:13.033125  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 148 candidate_status { last_received { term: 147 index: 123 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:13.033213  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 148 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 148: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:13.033730  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 148.
I20250114 03:12:13.033962  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:13.034070  4624 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 148 FOLLOWER]: Leader election won for term 148
I20250114 03:12:13.034176  4624 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 148 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:13.034221  4624 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 147.123, Last appended by leader: 123, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.034729  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 147 to 148, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 148 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:13.093152  4616 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 148 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:13.093258  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 148 FOLLOWER]: Advancing to term 149
I20250114 03:12:13.094015  4616 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.094198  4616 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 149 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:13.094444  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 149 candidate_status { last_received { term: 147 index: 123 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:13.094516  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 148 FOLLOWER]: Advancing to term 149
I20250114 03:12:13.094448  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 149 candidate_status { last_received { term: 147 index: 123 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:13.095161  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 149.
I20250114 03:12:13.095387  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 149 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:13.095537  4616 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 149 FOLLOWER]: Leader election won for term 149
I20250114 03:12:13.095611  4616 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 149 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:13.095674  4616 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 147.123, Last appended by leader: 123, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.096191  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 148 to 149, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 149 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:13.116468  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 149 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 148. Current term is 149. Ops: []
I20250114 03:12:13.116763  4624 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250114 03:12:13.116861  4624 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 94c6eb209d3e4298b92b940281c1fd5a log is divergent from this leader: its last log entry 149.124 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 123
I20250114 03:12:13.116884  4624 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 148 LEADER]: Stepping down as leader of term 148
I20250114 03:12:13.116915  4624 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 148 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:13.116978  4624 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 148.124, Last appended by leader: 124, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.117059  4624 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 148 FOLLOWER]: Advancing to term 149
I20250114 03:12:13.172345  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 149 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 148 index: 124. Preceding OpId from leader: term: 149 index: 124. (term mismatch)
I20250114 03:12:13.172431  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 123
I20250114 03:12:13.172454  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 148.124
I20250114 03:12:13.172477  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 149 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:13.172755  4616 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250114 03:12:13.173810  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 149 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 147 index: 123. Preceding OpId from leader: term: 149 index: 124. (index mismatch)
I20250114 03:12:13.174226  4628 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250114 03:12:13.275624  4625 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 149 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:13.275738  4625 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 149 FOLLOWER]: Advancing to term 150
I20250114 03:12:13.276543  4625 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.276737  4625 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 150 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:13.276959  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:13.276973  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:13.277223  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 150 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:13.277329  4625 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 150 FOLLOWER]: Leader election lost for term 150. Reason: could not achieve majority
I20250114 03:12:13.282946  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 150 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 149. Current term is 150. Ops: []
I20250114 03:12:13.283183  4616 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 149.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250114 03:12:13.283316  4616 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 149 LEADER]: Stepping down as leader of term 149
I20250114 03:12:13.283342  4616 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 149 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:13.283406  4616 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.283499  4616 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 149 FOLLOWER]: Advancing to term 150
I20250114 03:12:13.353576  4624 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 149 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:13.353681  4624 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 149 FOLLOWER]: Advancing to term 150
I20250114 03:12:13.354395  4624 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.354671  4624 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 150 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:13.354792  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:13.354854  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:13.354905  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 150 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 150: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:13.355463  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 150.
I20250114 03:12:13.355621  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 150 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:13.355712  4624 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 150 FOLLOWER]: Leader election won for term 150
I20250114 03:12:13.355760  4624 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 150 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:13.355830  4624 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.356302  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 149 to 150, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 150 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:13.384675  4625 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 150 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:13.384770  4625 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 150 FOLLOWER]: Advancing to term 151
I20250114 03:12:13.385450  4625 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.385627  4625 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 151 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:13.385890  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 151 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:13.385891  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 151 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:13.386018  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 150 FOLLOWER]: Advancing to term 151
I20250114 03:12:13.386638  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 151.
I20250114 03:12:13.386814  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 151 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:13.386904  4625 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 151 FOLLOWER]: Leader election won for term 151
I20250114 03:12:13.386961  4625 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 151 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:13.387013  4625 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.387492  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 150 to 151, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 151 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:13.447672  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 151 LEADER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 150. Current term is 151. Ops: []
I20250114 03:12:13.447960  4624 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250114 03:12:13.448071  4624 consensus_queue.cc:1225] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer 8a546775046844008af4967b5e3e1d14 log is divergent from this leader: its last log entry 151.125 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 124
I20250114 03:12:13.448117  4627 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 150 LEADER]: Stepping down as leader of term 150
I20250114 03:12:13.448158  4627 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 150 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:13.448220  4627 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 150.125, Last appended by leader: 125, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.448307  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 150 FOLLOWER]: Advancing to term 151
I20250114 03:12:13.468021  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 151 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 150 index: 125. Preceding OpId from leader: term: 151 index: 125. (term mismatch)
I20250114 03:12:13.468099  4273 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting all ops after (but not including) 124
I20250114 03:12:13.468112  4273 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: Aborting uncommitted NO_OP operation due to leader change: 150.125
I20250114 03:12:13.468129  4273 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 151 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:13.468362  4625 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250114 03:12:13.469295  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 151 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 149 index: 124. Preceding OpId from leader: term: 151 index: 125. (index mismatch)
I20250114 03:12:13.469539  4629 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250114 03:12:13.665890  4628 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 151 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:13.666000  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 151 FOLLOWER]: Advancing to term 152
I20250114 03:12:13.666783  4628 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.666972  4628 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 152 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:13.667201  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:13.667214  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:13.667415  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 152 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:13.667508  4628 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 152 FOLLOWER]: Leader election lost for term 152. Reason: could not achieve majority
I20250114 03:12:13.677911  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 152 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 151. Current term is 152. Ops: []
I20250114 03:12:13.678156  4625 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 151.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250114 03:12:13.678300  4625 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 151 LEADER]: Stepping down as leader of term 151
I20250114 03:12:13.678335  4625 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 151 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:13.678397  4625 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 151.125, Last appended by leader: 125, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.678483  4625 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 151 FOLLOWER]: Advancing to term 152
I20250114 03:12:13.689836  4627 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 151 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:13.689962  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 151 FOLLOWER]: Advancing to term 152
I20250114 03:12:13.690591  4627 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.690738  4627 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 152 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:13.690950  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:13.691017  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 152 FOLLOWER]: Leader election vote request: Denying vote to candidate 5eea63c188224be98e6888084ea42808 in current term 152: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:13.690956  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:13.691589  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 152.
I20250114 03:12:13.691771  4239 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 152 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:13.691855  4627 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 152 FOLLOWER]: Leader election won for term 152
I20250114 03:12:13.691905  4627 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 152 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:13.691958  4627 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 151.125, Last appended by leader: 125, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.692358  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 151 to 152, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 152 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:13.770651  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:13.771694  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 152 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 151 index: 125. Preceding OpId from leader: term: 152 index: 126. (index mismatch)
I20250114 03:12:13.771957  4627 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250114 03:12:13.772902  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 152 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 151 index: 125. Preceding OpId from leader: term: 152 index: 126. (index mismatch)
I20250114 03:12:13.773090  4627 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250114 03:12:13.873988  4628 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 152 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:13.874079  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 152 FOLLOWER]: Advancing to term 153
I20250114 03:12:13.874738  4628 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.874908  4628 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 153 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:13.875082  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:13.875140  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:13.875424  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 153 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:13.875525  4628 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 153 FOLLOWER]: Leader election lost for term 153. Reason: could not achieve majority
I20250114 03:12:13.892695  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 153 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 152. Current term is 153. Ops: []
I20250114 03:12:13.893026  4627 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 152.126, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250114 03:12:13.893133  4627 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 152 LEADER]: Stepping down as leader of term 152
I20250114 03:12:13.893164  4627 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 152 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:13.893226  4627 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 152.126, Last appended by leader: 126, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.893302  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 152 FOLLOWER]: Advancing to term 153
I20250114 03:12:13.971391  4625 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 152 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:13.971510  4625 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 152 FOLLOWER]: Advancing to term 153
I20250114 03:12:13.972196  4625 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.972414  4625 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 153 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:13.972657  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:13.972666  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 153 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:13.972787  4425 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 153 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 153: Already voted for candidate 94c6eb209d3e4298b92b940281c1fd5a in this term.
I20250114 03:12:13.973295  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 153.
I20250114 03:12:13.973536  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 153 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:13.973636  4625 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 153 FOLLOWER]: Leader election won for term 153
I20250114 03:12:13.973688  4625 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 153 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:13.973731  4625 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 152.126, Last appended by leader: 126, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:13.974298  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 152 to 153, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 153 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:14.010246  4628 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 153 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:14.010349  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 153 FOLLOWER]: Advancing to term 154
I20250114 03:12:14.011216  4628 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.011499  4628 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 154 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:14.011699  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 154 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:14.011699  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 154 candidate_status { last_received { term: 152 index: 126 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:14.011811  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 153 FOLLOWER]: Advancing to term 154
I20250114 03:12:14.012362  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 154.
I20250114 03:12:14.012558  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 154 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:14.012655  4628 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 154 FOLLOWER]: Leader election won for term 154
I20250114 03:12:14.012776  4628 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 154 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:14.012837  4628 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 152.126, Last appended by leader: 126, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.013350  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 153 to 154, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 154 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:14.077790  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 154 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 153. Current term is 154. Ops: []
I20250114 03:12:14.078120  4625 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250114 03:12:14.078229  4625 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 153 LEADER]: Stepping down as leader of term 153
I20250114 03:12:14.078248  4625 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 153 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:14.078303  4625 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 153.127, Last appended by leader: 127, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.078378  4625 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 153 FOLLOWER]: Advancing to term 154
I20250114 03:12:14.113802  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 154 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 153 index: 127. Preceding OpId from leader: term: 154 index: 127. (term mismatch)
I20250114 03:12:14.113906  4347 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting all ops after (but not including) 126
I20250114 03:12:14.113921  4347 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: Aborting uncommitted NO_OP operation due to leader change: 153.127
I20250114 03:12:14.113937  4347 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 154 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:14.114154  4628 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250114 03:12:14.115162  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 154 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 152 index: 126. Preceding OpId from leader: term: 154 index: 127. (index mismatch)
I20250114 03:12:14.115511  4631 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250114 03:12:14.479609  4627 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 154 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:14.479712  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 154 FOLLOWER]: Advancing to term 155
I20250114 03:12:14.480417  4627 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.480620  4627 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 155 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:14.480832  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:14.480830  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:14.481102  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 155 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:14.481184  4627 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 155 FOLLOWER]: Leader election lost for term 155. Reason: could not achieve majority
I20250114 03:12:14.502311  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 155 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 154. Current term is 155. Ops: []
I20250114 03:12:14.502573  4625 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 154 FOLLOWER]: Starting leader election (detected failure of leader 94c6eb209d3e4298b92b940281c1fd5a)
I20250114 03:12:14.502676  4625 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 154 FOLLOWER]: Advancing to term 155
I20250114 03:12:14.502595  4628 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 154.127, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250114 03:12:14.502746  4628 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 154 LEADER]: Stepping down as leader of term 154
I20250114 03:12:14.502782  4628 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 154 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:14.502835  4628 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 154.127, Last appended by leader: 127, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.502913  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 154 FOLLOWER]: Advancing to term 155
I20250114 03:12:14.503417  4625 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.503564  4625 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 155 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:14.503728  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:14.503800  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a546775046844008af4967b5e3e1d14 in current term 155: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:14.503741  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 155 candidate_status { last_received { term: 154 index: 127 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:14.504359  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 155.
I20250114 03:12:14.504520  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 155 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:14.504619  4625 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 155 FOLLOWER]: Leader election won for term 155
I20250114 03:12:14.504668  4625 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 155 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:14.504722  4625 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 154.127, Last appended by leader: 127, Current term: 155, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.505205  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 154 to 155, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 155 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:14.599766  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 155 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 154 index: 127. Preceding OpId from leader: term: 155 index: 128. (index mismatch)
I20250114 03:12:14.600080  4625 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250114 03:12:14.601336  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 155 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 154 index: 127. Preceding OpId from leader: term: 155 index: 128. (index mismatch)
I20250114 03:12:14.601558  4625 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250114 03:12:14.770934  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:14.802155  4627 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 155 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:14.802263  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 155 FOLLOWER]: Advancing to term 156
I20250114 03:12:14.802975  4627 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.803155  4627 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 156 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:14.803383  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:14.803383  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:14.803632  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 156 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5eea63c188224be98e6888084ea42808; no voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:14.803753  4627 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 156 FOLLOWER]: Leader election lost for term 156. Reason: could not achieve majority
I20250114 03:12:14.816423  4628 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 155 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:14.816521  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 155 FOLLOWER]: Advancing to term 156
I20250114 03:12:14.817279  4628 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.817438  4628 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 156 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:14.817662  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:14.817662  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 156 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:14.817770  4273 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 156 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 156: Already voted for candidate 5eea63c188224be98e6888084ea42808 in this term.
I20250114 03:12:14.817960  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 156 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14
I20250114 03:12:14.818061  4628 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 156 FOLLOWER]: Leader election lost for term 156. Reason: could not achieve majority
I20250114 03:12:14.823935  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 156 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 155. Current term is 156. Ops: []
I20250114 03:12:14.824221  4632 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 155.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250114 03:12:14.824295  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 156 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 155. Current term is 156. Ops: []
I20250114 03:12:14.824368  4626 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 155 LEADER]: Stepping down as leader of term 155
I20250114 03:12:14.824410  4626 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 155 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:14.824466  4626 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 155.128, Last appended by leader: 128, Current term: 155, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.824534  4632 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 155.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250114 03:12:14.824666  4626 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 155 FOLLOWER]: Advancing to term 156
I20250114 03:12:14.908224  4627 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 156 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:14.908335  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 156 FOLLOWER]: Advancing to term 157
I20250114 03:12:14.909018  4627 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.909181  4627 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 157 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:14.909426  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 157 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:14.909427  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 157 candidate_status { last_received { term: 155 index: 128 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:14.909543  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 156 FOLLOWER]: Advancing to term 157
I20250114 03:12:14.909554  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 156 FOLLOWER]: Advancing to term 157
I20250114 03:12:14.910168  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 157.
I20250114 03:12:14.910202  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 157.
I20250114 03:12:14.910365  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 
I20250114 03:12:14.910480  4627 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 157 FOLLOWER]: Leader election won for term 157
I20250114 03:12:14.910617  4627 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 157 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:14.910688  4627 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 155.128, Last appended by leader: 128, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:14.911137  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 155 to 157, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 157 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:15.006747  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 157 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 155 index: 128. Preceding OpId from leader: term: 157 index: 129. (index mismatch)
I20250114 03:12:15.007081  4627 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250114 03:12:15.008134  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 157 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 155 index: 128. Preceding OpId from leader: term: 157 index: 129. (index mismatch)
I20250114 03:12:15.008363  4627 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250114 03:12:15.108999  4626 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 157 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:15.109105  4626 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 157 FOLLOWER]: Advancing to term 158
I20250114 03:12:15.109807  4626 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.109998  4626 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 158 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:15.110214  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:15.110225  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:15.110476  4313 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 158 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a546775046844008af4967b5e3e1d14; no voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:15.110566  4626 raft_consensus.cc:2743] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 158 FOLLOWER]: Leader election lost for term 158. Reason: could not achieve majority
I20250114 03:12:15.124071  4347 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 158 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 157. Current term is 158. Ops: []
I20250114 03:12:15.124339  4627 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 157.129, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250114 03:12:15.124471  4627 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 157 LEADER]: Stepping down as leader of term 157
I20250114 03:12:15.124507  4627 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 157 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:15.124557  4627 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 157.129, Last appended by leader: 129, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.124663  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 157 FOLLOWER]: Advancing to term 158
I20250114 03:12:15.199326  4628 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 157 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:15.199434  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 157 FOLLOWER]: Advancing to term 158
I20250114 03:12:15.200125  4628 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.200322  4628 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 158 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:15.200563  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:15.200563  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 158 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:15.200680  4347 raft_consensus.cc:2388] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 158 FOLLOWER]: Leader election vote request: Denying vote to candidate 94c6eb209d3e4298b92b940281c1fd5a in current term 158: Already voted for candidate 8a546775046844008af4967b5e3e1d14 in this term.
I20250114 03:12:15.201243  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 158.
I20250114 03:12:15.201421  4387 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 158 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:15.201522  4628 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 158 FOLLOWER]: Leader election won for term 158
I20250114 03:12:15.201646  4628 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 158 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:15.201692  4628 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 157.129, Last appended by leader: 129, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.202316  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 157 to 158, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 158 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:15.254149  4626 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 158 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:15.254258  4626 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 158 FOLLOWER]: Advancing to term 159
I20250114 03:12:15.254937  4626 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.255097  4626 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 159 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675)
I20250114 03:12:15.255339  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 159 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:15.255344  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "8a546775046844008af4967b5e3e1d14" candidate_term: 159 candidate_status { last_received { term: 157 index: 129 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:15.255489  4273 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 158 FOLLOWER]: Advancing to term 159
I20250114 03:12:15.256101  4273 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 159 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a546775046844008af4967b5e3e1d14 in term 159.
I20250114 03:12:15.256278  4312 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [CANDIDATE]: Term 159 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 8a546775046844008af4967b5e3e1d14; no voters: 94c6eb209d3e4298b92b940281c1fd5a
I20250114 03:12:15.256377  4626 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 159 FOLLOWER]: Leader election won for term 159
I20250114 03:12:15.256440  4626 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 159 LEADER]: Becoming Leader. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:15.256498  4626 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 157.129, Last appended by leader: 129, Current term: 159, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.256973  4166 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 reported cstate change: term changed from 158 to 159, leader changed from 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131) to 8a546775046844008af4967b5e3e1d14 (127.4.10.130). New cstate: current_term: 159 leader_uuid: "8a546775046844008af4967b5e3e1d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:15.286247  4273 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 159 FOLLOWER]: Rejecting Update request from peer 94c6eb209d3e4298b92b940281c1fd5a for earlier term 158. Current term is 159. Ops: []
I20250114 03:12:15.286526  4628 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250114 03:12:15.286648  4628 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 158 LEADER]: Stepping down as leader of term 158
I20250114 03:12:15.286666  4628 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 158 LEADER]: Becoming Follower/Learner. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:15.286732  4628 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 158.130, Last appended by leader: 130, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.286824  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 158 FOLLOWER]: Advancing to term 159
I20250114 03:12:15.341737  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 159 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 158 index: 130. Preceding OpId from leader: term: 159 index: 130. (term mismatch)
I20250114 03:12:15.341827  4425 pending_rounds.cc:77] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting all ops after (but not including) 129
I20250114 03:12:15.341841  4425 pending_rounds.cc:99] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: Aborting uncommitted NO_OP operation due to leader change: 158.130
I20250114 03:12:15.341889  4425 raft_consensus.cc:2883] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 159 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:15.342180  4626 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250114 03:12:15.343266  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 159 FOLLOWER]: Refusing update from remote peer 8a546775046844008af4967b5e3e1d14: Log matching property violated. Preceding OpId in replica: term: 157 index: 129. Preceding OpId from leader: term: 159 index: 130. (index mismatch)
I20250114 03:12:15.343462  4626 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250114 03:12:15.444375  4627 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 159 FOLLOWER]: Starting leader election (detected failure of leader 8a546775046844008af4967b5e3e1d14)
I20250114 03:12:15.444481  4627 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 159 FOLLOWER]: Advancing to term 160
I20250114 03:12:15.445165  4627 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.445339  4627 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 160 election: Requested vote from peers 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131:39675), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:15.445576  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 160 candidate_status { last_received { term: 159 index: 130 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:15.445583  4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "5eea63c188224be98e6888084ea42808" candidate_term: 160 candidate_status { last_received { term: 159 index: 130 } } ignore_live_leader: false dest_uuid: "94c6eb209d3e4298b92b940281c1fd5a"
I20250114 03:12:15.445716  4425 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 159 FOLLOWER]: Advancing to term 160
I20250114 03:12:15.446375  4425 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eea63c188224be98e6888084ea42808 in term 160.
I20250114 03:12:15.446619  4237 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [CANDIDATE]: Term 160 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5eea63c188224be98e6888084ea42808, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 8a546775046844008af4967b5e3e1d14
I20250114 03:12:15.446740  4627 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 160 FOLLOWER]: Leader election won for term 160
I20250114 03:12:15.446827  4627 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 160 LEADER]: Becoming Leader. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:15.446897  4627 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 159.130, Last appended by leader: 130, Current term: 160, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.447396  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 reported cstate change: term changed from 159 to 160, leader changed from 8a546775046844008af4967b5e3e1d14 (127.4.10.130) to 5eea63c188224be98e6888084ea42808 (127.4.10.129). New cstate: current_term: 160 leader_uuid: "5eea63c188224be98e6888084ea42808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:15.462249  4425 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 160 FOLLOWER]: Rejecting Update request from peer 8a546775046844008af4967b5e3e1d14 for earlier term 159. Current term is 160. Ops: []
I20250114 03:12:15.462684  4626 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: INVALID_TERM, Last received: 159.130, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20250114 03:12:15.462812  4626 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 159 LEADER]: Stepping down as leader of term 159
I20250114 03:12:15.462833  4626 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 159 LEADER]: Becoming Follower/Learner. State: Replica: 8a546775046844008af4967b5e3e1d14, State: Running, Role: LEADER
I20250114 03:12:15.462885  4626 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 159.130, Last appended by leader: 130, Current term: 159, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.462972  4626 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 159 FOLLOWER]: Advancing to term 160
I20250114 03:12:15.524777  4347 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 160 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 159 index: 130. Preceding OpId from leader: term: 160 index: 131. (index mismatch)
I20250114 03:12:15.525068  4627 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20250114 03:12:15.526193  4425 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 160 FOLLOWER]: Refusing update from remote peer 5eea63c188224be98e6888084ea42808: Log matching property violated. Preceding OpId in replica: term: 159 index: 130. Preceding OpId from leader: term: 160 index: 131. (index mismatch)
I20250114 03:12:15.526403  4634 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20250114 03:12:15.627615  4628 raft_consensus.cc:491] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 160 FOLLOWER]: Starting leader election (detected failure of leader 5eea63c188224be98e6888084ea42808)
I20250114 03:12:15.627717  4628 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 160 FOLLOWER]: Advancing to term 161
I20250114 03:12:15.628433  4628 raft_consensus.cc:513] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.628620  4628 leader_election.cc:290] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 161 election: Requested vote from peers 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007), 8a546775046844008af4967b5e3e1d14 (127.4.10.130:35783)
I20250114 03:12:15.628891  4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 161 candidate_status { last_received { term: 160 index: 131 } } ignore_live_leader: false dest_uuid: "8a546775046844008af4967b5e3e1d14"
I20250114 03:12:15.628851  4273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508dc962d19543739571bcdc4c72574e" candidate_uuid: "94c6eb209d3e4298b92b940281c1fd5a" candidate_term: 161 candidate_status { last_received { term: 160 index: 131 } } ignore_live_leader: false dest_uuid: "5eea63c188224be98e6888084ea42808"
I20250114 03:12:15.628978  4347 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 160 FOLLOWER]: Advancing to term 161
I20250114 03:12:15.629581  4347 raft_consensus.cc:2463] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c6eb209d3e4298b92b940281c1fd5a in term 161.
I20250114 03:12:15.629704  4349 raft_consensus.cc:1235] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 161 FOLLOWER]: Rejecting Update request from peer 5eea63c188224be98e6888084ea42808 for earlier term 160. Current term is 161. Ops: []
I20250114 03:12:15.629777  4390 leader_election.cc:304] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [CANDIDATE]: Term 161 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a546775046844008af4967b5e3e1d14, 94c6eb209d3e4298b92b940281c1fd5a; no voters: 5eea63c188224be98e6888084ea42808
I20250114 03:12:15.629920  4628 raft_consensus.cc:2798] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 161 FOLLOWER]: Leader election won for term 161
I20250114 03:12:15.629935  4634 consensus_queue.cc:1046] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: INVALID_TERM, Last received: 160.131, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20250114 03:12:15.629999  4628 raft_consensus.cc:695] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 161 LEADER]: Becoming Leader. State: Replica: 94c6eb209d3e4298b92b940281c1fd5a, State: Running, Role: LEADER
I20250114 03:12:15.630059  4634 raft_consensus.cc:3049] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 160 LEADER]: Stepping down as leader of term 160
I20250114 03:12:15.630091  4634 raft_consensus.cc:738] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 160 LEADER]: Becoming Follower/Learner. State: Replica: 5eea63c188224be98e6888084ea42808, State: Running, Role: LEADER
I20250114 03:12:15.630074  4628 consensus_queue.cc:237] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 160.131, Last appended by leader: 131, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.630189  4634 consensus_queue.cc:260] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 160.131, Last appended by leader: 131, Current term: 160, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } }
I20250114 03:12:15.630276  4634 raft_consensus.cc:3054] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 160 FOLLOWER]: Advancing to term 161
I20250114 03:12:15.630604  4167 catalog_manager.cc:5526] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a reported cstate change: term changed from 160 to 161, leader changed from 5eea63c188224be98e6888084ea42808 (127.4.10.129) to 94c6eb209d3e4298b92b940281c1fd5a (127.4.10.131). New cstate: current_term: 161 leader_uuid: "94c6eb209d3e4298b92b940281c1fd5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c6eb209d3e4298b92b940281c1fd5a" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 39675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:15.739025  4349 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 161 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 160 index: 131. Preceding OpId from leader: term: 161 index: 132. (index mismatch)
I20250114 03:12:15.739399  4628 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a546775046844008af4967b5e3e1d14" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20250114 03:12:15.740751  4273 raft_consensus.cc:1270] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 161 FOLLOWER]: Refusing update from remote peer 94c6eb209d3e4298b92b940281c1fd5a: Log matching property violated. Preceding OpId in replica: term: 160 index: 131. Preceding OpId from leader: term: 161 index: 132. (index mismatch)
I20250114 03:12:15.741034  4635 consensus_queue.cc:1035] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eea63c188224be98e6888084ea42808" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20250114 03:12:15.771219  4138 timestamp_advancement-itest.cc:261] GCing logs...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:261: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:326: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20250114 03:12:15.771472  4138 tablet_server.cc:178] TabletServer@127.4.10.129:0 shutting down...
I20250114 03:12:15.819656  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:15.819854  4138 tablet_replica.cc:331] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808: stopping tablet replica
I20250114 03:12:15.819988  4138 raft_consensus.cc:2238] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 161 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:15.820058  4138 raft_consensus.cc:2267] T 508dc962d19543739571bcdc4c72574e P 5eea63c188224be98e6888084ea42808 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.833340  4138 tablet_server.cc:195] TabletServer@127.4.10.129:0 shutdown complete.
I20250114 03:12:15.834694  4138 tablet_server.cc:178] TabletServer@127.4.10.130:0 shutting down...
I20250114 03:12:15.848742  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:15.848940  4138 tablet_replica.cc:331] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14: stopping tablet replica
I20250114 03:12:15.849030  4138 raft_consensus.cc:2238] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 161 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:15.849072  4138 raft_consensus.cc:2267] T 508dc962d19543739571bcdc4c72574e P 8a546775046844008af4967b5e3e1d14 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.861245  4138 tablet_server.cc:195] TabletServer@127.4.10.130:0 shutdown complete.
I20250114 03:12:15.862211  4138 tablet_server.cc:178] TabletServer@127.4.10.131:0 shutting down...
W20250114 03:12:15.865794  4387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.10.129:43007: connect: Connection refused (error 111)
W20250114 03:12:15.866195  4387 consensus_peers.cc:487] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a -> Peer 5eea63c188224be98e6888084ea42808 (127.4.10.129:43007): Couldn't send request to peer 5eea63c188224be98e6888084ea42808. Status: Network error: Client connection negotiation failed: client connection to 127.4.10.129:43007: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:12:15.869115  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:15.869215  4138 tablet_replica.cc:331] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a: stopping tablet replica
I20250114 03:12:15.869278  4138 raft_consensus.cc:2238] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 161 LEADER]: Raft consensus shutting down.
I20250114 03:12:15.869346  4138 raft_consensus.cc:2267] T 508dc962d19543739571bcdc4c72574e P 94c6eb209d3e4298b92b940281c1fd5a [term 161 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.881314  4138 tablet_server.cc:195] TabletServer@127.4.10.131:0 shutdown complete.
I20250114 03:12:15.882349  4138 master.cc:537] Master@127.4.10.190:45173 shutting down...
I20250114 03:12:15.994648  4138 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:12:15.994760  4138 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:15.994792  4138 tablet_replica.cc:331] T 00000000000000000000000000000000 P f2082d9a67064f508e2f7e78d7dd65fa: stopping tablet replica
I20250114 03:12:16.006979  4138 master.cc:559] Master@127.4.10.190:45173 shutdown complete.
I20250114 03:12:16.008780  4138 test_util.cc:181] -----------------------------------------------
I20250114 03:12:16.008826  4138 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1736824303196632-4138-0
[  FAILED  ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (32808 ms)
[ RUN      ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20250114 03:12:16.009305  4138 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.10.190:34423
I20250114 03:12:16.009459  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:16.011945  4640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.012815  4643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.013442  4641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:16.014051  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:16.014144  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:16.014179  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:16.014194  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824336014193 us; error 0 us; skew 500 ppm
I20250114 03:12:16.014885  4138 webserver.cc:458] Webserver started at http://127.4.10.190:43371/ using document root <none> and password file <none>
I20250114 03:12:16.014983  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:16.015038  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:16.015086  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:16.015374  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/master-0-root/instance:
uuid: "062e89d1e061457681905c352f30f818"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.016350  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:16.016842  4648 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.016966  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.017019  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/master-0-root
uuid: "062e89d1e061457681905c352f30f818"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.017063  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:16.023432  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:16.023591  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:16.026196  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.190:34423
I20250114 03:12:16.028003  4700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.190:34423 every 8 connection(s)
I20250114 03:12:16.028162  4701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:16.029174  4701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818: Bootstrap starting.
I20250114 03:12:16.029500  4701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:16.029983  4701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818: No bootstrap required, opened a new log
I20250114 03:12:16.030103  4701 raft_consensus.cc:357] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e89d1e061457681905c352f30f818" member_type: VOTER }
I20250114 03:12:16.030159  4701 raft_consensus.cc:383] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:16.030174  4701 raft_consensus.cc:738] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062e89d1e061457681905c352f30f818, State: Initialized, Role: FOLLOWER
I20250114 03:12:16.030227  4701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e89d1e061457681905c352f30f818" member_type: VOTER }
I20250114 03:12:16.030265  4701 raft_consensus.cc:397] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:12:16.030294  4701 raft_consensus.cc:491] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:12:16.030335  4701 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:16.030814  4701 raft_consensus.cc:513] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e89d1e061457681905c352f30f818" member_type: VOTER }
I20250114 03:12:16.030866  4701 leader_election.cc:304] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 062e89d1e061457681905c352f30f818; no voters: 
I20250114 03:12:16.030963  4701 leader_election.cc:290] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20250114 03:12:16.031028  4704 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:12:16.031149  4701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:12:16.031185  4704 raft_consensus.cc:695] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 1 LEADER]: Becoming Leader. State: Replica: 062e89d1e061457681905c352f30f818, State: Running, Role: LEADER
I20250114 03:12:16.031308  4704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e89d1e061457681905c352f30f818" member_type: VOTER }
I20250114 03:12:16.031550  4705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "062e89d1e061457681905c352f30f818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e89d1e061457681905c352f30f818" member_type: VOTER } }
I20250114 03:12:16.031558  4706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 062e89d1e061457681905c352f30f818. Latest consensus state: current_term: 1 leader_uuid: "062e89d1e061457681905c352f30f818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e89d1e061457681905c352f30f818" member_type: VOTER } }
I20250114 03:12:16.031731  4706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:16.031658  4705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:16.031957  4713 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:12:16.032115  4713 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:12:16.032646  4713 catalog_manager.cc:1348] Generated new cluster ID: b4df7b17dba64dcc88410ae83939ad26
I20250114 03:12:16.032694  4713 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:12:16.033919  4138 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 03:12:16.053278  4713 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:12:16.053714  4713 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:12:16.058007  4713 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818: Generated new TSK 0
I20250114 03:12:16.058071  4713 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:12:16.065742  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:16.068053  4722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.068969  4725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.069657  4723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:16.070252  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:16.070355  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:16.070389  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:16.070410  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824336070409 us; error 0 us; skew 500 ppm
I20250114 03:12:16.071056  4138 webserver.cc:458] Webserver started at http://127.4.10.129:38617/ using document root <none> and password file <none>
I20250114 03:12:16.071144  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:16.071192  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:16.071246  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:16.071532  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root/instance:
uuid: "6a4b32f73dce497a86df7c103c209107"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.072436  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:16.072923  4730 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.073063  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.073108  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root
uuid: "6a4b32f73dce497a86df7c103c209107"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.073153  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:16.081244  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:16.081418  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:16.081557  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:16.081753  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:16.081806  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.081840  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:16.081889  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.084812  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.129:41253
I20250114 03:12:16.084839  4794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.129:41253 every 8 connection(s)
I20250114 03:12:16.085152  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:16.087092  4799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.087704  4800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.088388  4802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:16.088428  4795 heartbeater.cc:346] Connected to a master server at 127.4.10.190:34423
I20250114 03:12:16.088475  4795 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:16.088593  4795 heartbeater.cc:510] Master 127.4.10.190:34423 requested a full tablet report, sending...
I20250114 03:12:16.088840  4665 ts_manager.cc:194] Registered new tserver with Master: 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:16.089272  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:16.089392  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:16.089422  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:16.089440  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824336089440 us; error 0 us; skew 500 ppm
I20250114 03:12:16.089442  4665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38798
I20250114 03:12:16.090101  4138 webserver.cc:458] Webserver started at http://127.4.10.130:33475/ using document root <none> and password file <none>
I20250114 03:12:16.090169  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:16.090204  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:16.090237  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:16.090469  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-1-root/instance:
uuid: "2c99007133884f7890f978a641b3a902"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.091324  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:16.091833  4807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.091981  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.092037  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-1-root
uuid: "2c99007133884f7890f978a641b3a902"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.092077  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:16.108407  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:16.108599  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:16.108736  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:16.108955  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:16.108985  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.109011  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:16.109040  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.112051  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.130:38443
I20250114 03:12:16.112089  4870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.130:38443 every 8 connection(s)
I20250114 03:12:16.112413  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:16.114375  4874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.114935  4875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:16.115571  4877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:16.115674  4871 heartbeater.cc:346] Connected to a master server at 127.4.10.190:34423
I20250114 03:12:16.115754  4871 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:16.115871  4871 heartbeater.cc:510] Master 127.4.10.190:34423 requested a full tablet report, sending...
I20250114 03:12:16.116113  4665 ts_manager.cc:194] Registered new tserver with Master: 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:16.116442  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:16.116534  4665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38810
I20250114 03:12:16.116605  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:16.116647  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:16.116670  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824336116670 us; error 0 us; skew 500 ppm
I20250114 03:12:16.117384  4138 webserver.cc:458] Webserver started at http://127.4.10.131:32977/ using document root <none> and password file <none>
I20250114 03:12:16.117472  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:16.117516  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:16.117564  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:16.117897  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-2-root/instance:
uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.118796  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:16.119222  4882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.119343  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.119390  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-2-root
uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:16.119436  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:16.131222  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:16.131471  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:16.131629  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:16.131851  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:16.131882  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.131913  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:16.131943  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.135246  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.131:33927
I20250114 03:12:16.135282  4945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.131:33927 every 8 connection(s)
I20250114 03:12:16.138132  4946 heartbeater.cc:346] Connected to a master server at 127.4.10.190:34423
I20250114 03:12:16.138188  4946 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:16.138291  4946 heartbeater.cc:510] Master 127.4.10.190:34423 requested a full tablet report, sending...
I20250114 03:12:16.138533  4665 ts_manager.cc:194] Registered new tserver with Master: 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927)
I20250114 03:12:16.138748  4138 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003314126s
I20250114 03:12:16.138857  4665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38818
I20250114 03:12:16.145807  4138 test_util.cc:274] Using random seed: -308814452
I20250114 03:12:16.149817  4665 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38844:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20250114 03:12:16.149986  4665 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:12:16.154768  4835 tablet_service.cc:1467] Processing CreateTablet for tablet 69f16283f4bb432589185fa78ca4dbc9 (DEFAULT_TABLE table=test-workload [id=ac32fc2b5ccc4c7ba53bb1a2e0c46f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:12:16.154788  4910 tablet_service.cc:1467] Processing CreateTablet for tablet 69f16283f4bb432589185fa78ca4dbc9 (DEFAULT_TABLE table=test-workload [id=ac32fc2b5ccc4c7ba53bb1a2e0c46f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:12:16.154906  4835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f16283f4bb432589185fa78ca4dbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:16.154950  4758 tablet_service.cc:1467] Processing CreateTablet for tablet 69f16283f4bb432589185fa78ca4dbc9 (DEFAULT_TABLE table=test-workload [id=ac32fc2b5ccc4c7ba53bb1a2e0c46f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:12:16.154907  4910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f16283f4bb432589185fa78ca4dbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:16.155031  4758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f16283f4bb432589185fa78ca4dbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:16.156175  4967 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: Bootstrap starting.
I20250114 03:12:16.156463  4968 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Bootstrap starting.
I20250114 03:12:16.156605  4967 tablet_bootstrap.cc:654] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:16.157142  4968 tablet_bootstrap.cc:654] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:16.157349  4967 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: No bootstrap required, opened a new log
I20250114 03:12:16.157402  4967 ts_tablet_manager.cc:1397] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:16.157526  4969 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Bootstrap starting.
I20250114 03:12:16.157570  4967 raft_consensus.cc:357] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.157644  4967 raft_consensus.cc:383] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:16.157665  4967 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b1d948fc9846fa99a040f4cc2c02b4, State: Initialized, Role: FOLLOWER
I20250114 03:12:16.157707  4968 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: No bootstrap required, opened a new log
I20250114 03:12:16.157771  4968 ts_tablet_manager.cc:1397] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:16.157734  4967 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.157965  4968 raft_consensus.cc:357] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.158023  4967 ts_tablet_manager.cc:1428] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:16.158063  4969 tablet_bootstrap.cc:654] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:16.158059  4946 heartbeater.cc:502] Master 127.4.10.190:34423 was elected leader, sending a full tablet report...
I20250114 03:12:16.158087  4968 raft_consensus.cc:383] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:16.158445  4968 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c99007133884f7890f978a641b3a902, State: Initialized, Role: FOLLOWER
I20250114 03:12:16.158607  4969 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: No bootstrap required, opened a new log
I20250114 03:12:16.158576  4968 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.158682  4969 ts_tablet_manager.cc:1397] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Time spent bootstrapping tablet: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:16.158869  4968 ts_tablet_manager.cc:1428] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:16.158892  4871 heartbeater.cc:502] Master 127.4.10.190:34423 was elected leader, sending a full tablet report...
I20250114 03:12:16.158915  4969 raft_consensus.cc:357] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.158974  4969 raft_consensus.cc:383] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:16.159006  4969 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4b32f73dce497a86df7c103c209107, State: Initialized, Role: FOLLOWER
I20250114 03:12:16.159061  4969 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.159227  4969 ts_tablet_manager.cc:1428] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:16.159238  4795 heartbeater.cc:502] Master 127.4.10.190:34423 was elected leader, sending a full tablet report...
I20250114 03:12:16.176779  4974 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:12:16.176877  4974 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.177057  4974 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:16.179878  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107" is_pre_election: true
I20250114 03:12:16.179924  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" is_pre_election: true
I20250114 03:12:16.179977  4768 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 0.
I20250114 03:12:16.180023  4920 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 0.
I20250114 03:12:16.180191  4809 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902; no voters: 
I20250114 03:12:16.180284  4974 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:12:16.180325  4974 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:16.180354  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:16.180982  4974 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.181172  4974 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 1 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:16.181231  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:16.181293  4920 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:16.181324  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:16.181396  4768 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:16.181975  4920 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 1.
I20250114 03:12:16.182116  4768 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 1.
I20250114 03:12:16.182138  4809 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902; no voters: 
I20250114 03:12:16.182231  4974 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:12:16.182377  4974 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 1 LEADER]: Becoming Leader. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:16.182425  4974 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.182946  4665 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c99007133884f7890f978a641b3a902 (127.4.10.130). New cstate: current_term: 1 leader_uuid: "2c99007133884f7890f978a641b3a902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:16.191439  4768 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 1 FOLLOWER]: Refusing update from remote peer 2c99007133884f7890f978a641b3a902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:12:16.191473  4920 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 1 FOLLOWER]: Refusing update from remote peer 2c99007133884f7890f978a641b3a902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:12:16.191670  4974 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:12:16.191753  4977 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:12:16.193361  4984 mvcc.cc:204] Tried to move back new op lower bound from 7114032481040117760 to 7114032481003634688. Current Snapshot: MvccSnapshot[applied={T|T < 7114032481040117760 or (T in {7114032481040117760})}]
I20250114 03:12:16.420727  4975 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c99007133884f7890f978a641b3a902)
I20250114 03:12:16.420835  4975 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:12:16.421640  4975 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.421892  4975 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 2 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:16.424490  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:16.424516  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:16.424767  4733 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4b32f73dce497a86df7c103c209107; no voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902
I20250114 03:12:16.424881  4975 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 03:12:16.430241  4768 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 2 FOLLOWER]: Rejecting Update request from peer 2c99007133884f7890f978a641b3a902 for earlier term 1. Current term is 2. Ops: []
I20250114 03:12:16.430454  4974 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 }, Status: INVALID_TERM, Last received: 1.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 03:12:16.450663  4974 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:12:16.450755  4974 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:16.450827  4974 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.450943  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:12:16.509604  4972 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c99007133884f7890f978a641b3a902)
I20250114 03:12:16.509717  4972 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:12:16.510474  4972 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.510682  4972 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 2 election: Requested vote from peers 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:16.514326  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:16.514344  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:16.514446  4768 raft_consensus.cc:2388] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 01b1d948fc9846fa99a040f4cc2c02b4 in current term 2: Already voted for candidate 6a4b32f73dce497a86df7c103c209107 in this term.
I20250114 03:12:16.515118  4845 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b1d948fc9846fa99a040f4cc2c02b4 in term 2.
I20250114 03:12:16.515359  4885 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902; no voters: 6a4b32f73dce497a86df7c103c209107
I20250114 03:12:16.515497  4972 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:12:16.515635  4972 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 2 LEADER]: Becoming Leader. State: Replica: 01b1d948fc9846fa99a040f4cc2c02b4, State: Running, Role: LEADER
I20250114 03:12:16.515724  4972 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.516198  4663 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 reported cstate change: term changed from 1 to 2, leader changed from 2c99007133884f7890f978a641b3a902 (127.4.10.130) to 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131). New cstate: current_term: 2 leader_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:16.540328  4975 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:16.540421  4975 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:12:16.541060  4975 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.541231  4975 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 3 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:16.541440  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 3 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:16.541471  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 3 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:16.541536  4845 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:12:16.542145  4845 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4b32f73dce497a86df7c103c209107 in term 3.
I20250114 03:12:16.542338  4733 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c99007133884f7890f978a641b3a902, 6a4b32f73dce497a86df7c103c209107; no voters: 01b1d948fc9846fa99a040f4cc2c02b4
I20250114 03:12:16.542433  4975 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 03:12:16.542588  4975 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 3 LEADER]: Becoming Leader. State: Replica: 6a4b32f73dce497a86df7c103c209107, State: Running, Role: LEADER
I20250114 03:12:16.542670  4975 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.575029  4665 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 reported cstate change: term changed from 2 to 3, leader changed from 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131) to 6a4b32f73dce497a86df7c103c209107 (127.4.10.129). New cstate: current_term: 3 leader_uuid: "6a4b32f73dce497a86df7c103c209107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:16.601729  4845 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 3 FOLLOWER]: Rejecting Update request from peer 01b1d948fc9846fa99a040f4cc2c02b4 for earlier term 2. Current term is 3. Ops: []
I20250114 03:12:16.602087  4972 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 03:12:16.621507  4845 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 3 FOLLOWER]: Refusing update from remote peer 6a4b32f73dce497a86df7c103c209107: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20250114 03:12:16.621810  4975 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 03:12:16.628340  4972 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 2 LEADER]: Stepping down as leader of term 2
I20250114 03:12:16.628434  4972 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01b1d948fc9846fa99a040f4cc2c02b4, State: Running, Role: LEADER
I20250114 03:12:16.628531  4972 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.19, Last appended by leader: 19, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.628621  4972 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:12:16.653043  4920 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 3 FOLLOWER]: Refusing update from remote peer 6a4b32f73dce497a86df7c103c209107: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 3 index: 19. (term mismatch)
I20250114 03:12:16.653124  4920 pending_rounds.cc:77] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: Aborting all ops after (but not including) 18
I20250114 03:12:16.653146  4920 pending_rounds.cc:99] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: Aborting uncommitted NO_OP operation due to leader change: 2.19
I20250114 03:12:16.653167  4920 raft_consensus.cc:2883] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:16.653410  4975 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 03:12:16.986332  4974 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6a4b32f73dce497a86df7c103c209107)
I20250114 03:12:16.986440  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:12:16.988794  4974 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:16.988953  4845 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 4 FOLLOWER]: Rejecting Update request from peer 6a4b32f73dce497a86df7c103c209107 for earlier term 3. Current term is 4. Ops: []
I20250114 03:12:16.989032  4974 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 4 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:16.989219  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:16.989270  5005 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 }, Status: INVALID_TERM, Last received: 3.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 03:12:16.989226  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:16.989535  4808 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c99007133884f7890f978a641b3a902; no voters: 01b1d948fc9846fa99a040f4cc2c02b4, 6a4b32f73dce497a86df7c103c209107
I20250114 03:12:16.989635  4974 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 03:12:17.070492  4845 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 4 FOLLOWER]: Rejecting Update request from peer 6a4b32f73dce497a86df7c103c209107 for earlier term 3. Current term is 4. Ops: []
I20250114 03:12:17.070811  5006 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 }, Status: INVALID_TERM, Last received: 3.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 03:12:17.075457  5005 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 3 LEADER]: Stepping down as leader of term 3
I20250114 03:12:17.075518  5005 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6a4b32f73dce497a86df7c103c209107, State: Running, Role: LEADER
I20250114 03:12:17.075603  5005 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.075747  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:12:17.120801  4972 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6a4b32f73dce497a86df7c103c209107)
I20250114 03:12:17.120906  4972 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:12:17.121596  4972 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.121778  4972 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 4 election: Requested vote from peers 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:17.122002  4974 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:17.122028  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:17.122103  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 4 FOLLOWER]: Advancing to term 5
I20250114 03:12:17.122031  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:17.122821  4768 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b1d948fc9846fa99a040f4cc2c02b4 in term 4.
I20250114 03:12:17.122821  4974 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.122990  4845 raft_consensus.cc:2363] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01b1d948fc9846fa99a040f4cc2c02b4 for earlier term 4. Current term is 5.
I20250114 03:12:17.123019  4974 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 5 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:17.123163  4885 leader_election.cc:400] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 4 election: Vote denied by peer 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443) with higher term. Message: Invalid argument: T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01b1d948fc9846fa99a040f4cc2c02b4 for earlier term 4. Current term is 5.
I20250114 03:12:17.123200  4885 leader_election.cc:403] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250114 03:12:17.123207  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 5 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:17.123286  4972 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 4 FOLLOWER]: Advancing to term 5
I20250114 03:12:17.123296  4768 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 4 FOLLOWER]: Advancing to term 5
I20250114 03:12:17.123349  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 5 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:17.123916  4768 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 5.
I20250114 03:12:17.123916  4972 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443) with higher term. Message: Invalid argument: T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01b1d948fc9846fa99a040f4cc2c02b4 for earlier term 4. Current term is 5.
I20250114 03:12:17.124107  4808 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c99007133884f7890f978a641b3a902, 6a4b32f73dce497a86df7c103c209107; no voters: 
I20250114 03:12:17.124197  4974 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 03:12:17.124300  4974 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 LEADER]: Becoming Leader. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:17.124349  4974 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.124789  4920 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 5.
I20250114 03:12:17.124761  4665 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 reported cstate change: term changed from 3 to 5, leader changed from 6a4b32f73dce497a86df7c103c209107 (127.4.10.129) to 2c99007133884f7890f978a641b3a902 (127.4.10.130). New cstate: current_term: 5 leader_uuid: "2c99007133884f7890f978a641b3a902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:17.216797  4768 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 5 FOLLOWER]: Refusing update from remote peer 2c99007133884f7890f978a641b3a902: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 5 index: 20. (index mismatch)
I20250114 03:12:17.217074  4974 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 03:12:17.232318  4920 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 5 FOLLOWER]: Refusing update from remote peer 2c99007133884f7890f978a641b3a902: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 5 index: 20. (index mismatch)
I20250114 03:12:17.232695  4974 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 03:12:17.333806  4972 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2c99007133884f7890f978a641b3a902)
I20250114 03:12:17.333925  4972 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:12:17.334584  4972 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.334764  4972 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 6 election: Requested vote from peers 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:17.334977  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:17.334973  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:17.335206  4885 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4; no voters: 2c99007133884f7890f978a641b3a902, 6a4b32f73dce497a86df7c103c209107
I20250114 03:12:17.335296  4972 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250114 03:12:17.348737  4920 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 6 FOLLOWER]: Rejecting Update request from peer 2c99007133884f7890f978a641b3a902 for earlier term 5. Current term is 6. Ops: []
I20250114 03:12:17.349030  5007 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: INVALID_TERM, Last received: 5.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 03:12:17.435704  4974 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 03:12:17.435792  4974 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:17.435868  4974 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.436004  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:12:17.449239  4972 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:17.449328  4972 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:12:17.449999  4972 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.450170  4972 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 7 election: Requested vote from peers 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:17.450381  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 7 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:17.450431  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 7 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:17.450526  4845 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:12:17.451099  4845 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b1d948fc9846fa99a040f4cc2c02b4 in term 7.
I20250114 03:12:17.451256  4885 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902; no voters: 6a4b32f73dce497a86df7c103c209107
I20250114 03:12:17.451346  4972 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 7 FOLLOWER]: Leader election won for term 7
I20250114 03:12:17.451485  4972 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 7 LEADER]: Becoming Leader. State: Replica: 01b1d948fc9846fa99a040f4cc2c02b4, State: Running, Role: LEADER
I20250114 03:12:17.451535  4972 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.451978  4665 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 reported cstate change: term changed from 5 to 7, leader changed from 2c99007133884f7890f978a641b3a902 (127.4.10.130) to 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131). New cstate: current_term: 7 leader_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:17.484680  5005 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 2c99007133884f7890f978a641b3a902)
I20250114 03:12:17.484791  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:12:17.485509  5005 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.485689  5005 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 6 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:17.485950  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:17.486013  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:17.486023  4845 raft_consensus.cc:2363] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a4b32f73dce497a86df7c103c209107 for earlier term 6. Current term is 7.
I20250114 03:12:17.486281  4733 leader_election.cc:400] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 6 election: Vote denied by peer 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443) with higher term. Message: Invalid argument: T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a4b32f73dce497a86df7c103c209107 for earlier term 6. Current term is 7.
I20250114 03:12:17.486325  4733 leader_election.cc:403] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250114 03:12:17.486406  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:12:17.486935  5005 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443) with higher term. Message: Invalid argument: T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a4b32f73dce497a86df7c103c209107 for earlier term 6. Current term is 7.
I20250114 03:12:17.543535  4845 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 7 FOLLOWER]: Refusing update from remote peer 01b1d948fc9846fa99a040f4cc2c02b4: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
I20250114 03:12:17.543843  4972 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 03:12:17.569677  4768 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 7 FOLLOWER]: Refusing update from remote peer 01b1d948fc9846fa99a040f4cc2c02b4: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
I20250114 03:12:17.569980  4972 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 03:12:17.671428  5005 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 01b1d948fc9846fa99a040f4cc2c02b4)
I20250114 03:12:17.671527  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:12:17.672178  5005 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.672344  5005 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 8 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:17.672574  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:17.672577  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:17.672809  4732 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4b32f73dce497a86df7c103c209107; no voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902
I20250114 03:12:17.672909  5005 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250114 03:12:17.675179  4768 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 8 FOLLOWER]: Rejecting Update request from peer 01b1d948fc9846fa99a040f4cc2c02b4 for earlier term 7. Current term is 8. Ops: []
I20250114 03:12:17.675478  5016 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 }, Status: INVALID_TERM, Last received: 7.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250114 03:12:17.767428  5015 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 7 LEADER]: Stepping down as leader of term 7
I20250114 03:12:17.767526  5015 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 01b1d948fc9846fa99a040f4cc2c02b4, State: Running, Role: LEADER
I20250114 03:12:17.767612  5015 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 7.21, Last appended by leader: 21, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.767730  5015 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:12:17.790822  5005 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:17.790938  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 8 FOLLOWER]: Advancing to term 9
I20250114 03:12:17.791600  5005 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.791780  5005 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 9 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:17.792042  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 9 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:17.792042  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 9 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:17.792197  4920 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 8 FOLLOWER]: Advancing to term 9
I20250114 03:12:17.792857  4920 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4b32f73dce497a86df7c103c209107 in term 9.
I20250114 03:12:17.793049  4732 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 6a4b32f73dce497a86df7c103c209107; no voters: 2c99007133884f7890f978a641b3a902
I20250114 03:12:17.793155  5005 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 9 FOLLOWER]: Leader election won for term 9
I20250114 03:12:17.793277  5005 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 9 LEADER]: Becoming Leader. State: Replica: 6a4b32f73dce497a86df7c103c209107, State: Running, Role: LEADER
I20250114 03:12:17.793433  5005 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 7.21, Last appended by leader: 21, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.793952  4665 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 reported cstate change: term changed from 7 to 9, leader changed from 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131) to 6a4b32f73dce497a86df7c103c209107 (127.4.10.129). New cstate: current_term: 9 leader_uuid: "6a4b32f73dce497a86df7c103c209107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:17.838450  4974 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 01b1d948fc9846fa99a040f4cc2c02b4)
I20250114 03:12:17.838550  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:12:17.839197  4974 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:17.839375  4974 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 8 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:17.839578  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:17.839588  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:17.839676  4920 raft_consensus.cc:2363] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c99007133884f7890f978a641b3a902 for earlier term 8. Current term is 9.
I20250114 03:12:17.839844  4809 leader_election.cc:400] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 8 election: Vote denied by peer 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927) with higher term. Message: Invalid argument: T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c99007133884f7890f978a641b3a902 for earlier term 8. Current term is 9.
I20250114 03:12:17.839890  4809 leader_election.cc:403] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250114 03:12:17.839956  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 8 FOLLOWER]: Advancing to term 9
I20250114 03:12:17.840523  4974 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927) with higher term. Message: Invalid argument: T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c99007133884f7890f978a641b3a902 for earlier term 8. Current term is 9.
I20250114 03:12:17.904469  4920 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 9 FOLLOWER]: Refusing update from remote peer 6a4b32f73dce497a86df7c103c209107: Log matching property violated. Preceding OpId in replica: term: 7 index: 21. Preceding OpId from leader: term: 9 index: 22. (index mismatch)
I20250114 03:12:17.904800  5005 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250114 03:12:17.916572  4845 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 9 FOLLOWER]: Refusing update from remote peer 6a4b32f73dce497a86df7c103c209107: Log matching property violated. Preceding OpId in replica: term: 7 index: 21. Preceding OpId from leader: term: 9 index: 22. (index mismatch)
I20250114 03:12:17.916886  5005 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250114 03:12:18.006292  5015 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6a4b32f73dce497a86df7c103c209107)
I20250114 03:12:18.006381  5015 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 9 FOLLOWER]: Advancing to term 10
I20250114 03:12:18.007047  5015 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.007197  5015 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 10 election: Requested vote from peers 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:18.007386  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:18.007386  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:18.007674  4885 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4; no voters: 2c99007133884f7890f978a641b3a902, 6a4b32f73dce497a86df7c103c209107
I20250114 03:12:18.007763  5015 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250114 03:12:18.024011  4920 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 10 FOLLOWER]: Rejecting Update request from peer 6a4b32f73dce497a86df7c103c209107 for earlier term 9. Current term is 10. Ops: []
I20250114 03:12:18.024323  5017 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: INVALID_TERM, Last received: 9.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 03:12:18.072592  5005 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 9 LEADER]: Stepping down as leader of term 9
I20250114 03:12:18.072693  5005 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6a4b32f73dce497a86df7c103c209107, State: Running, Role: LEADER
I20250114 03:12:18.072763  5005 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 9.22, Last appended by leader: 22, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.072901  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 9 FOLLOWER]: Advancing to term 10
I20250114 03:12:18.110049  4974 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6a4b32f73dce497a86df7c103c209107)
I20250114 03:12:18.110164  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 9 FOLLOWER]: Advancing to term 10
I20250114 03:12:18.110888  4974 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.111063  4974 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 10 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:18.111301  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:18.111315  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:18.111430  4920 raft_consensus.cc:2388] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c99007133884f7890f978a641b3a902 in current term 10: Already voted for candidate 01b1d948fc9846fa99a040f4cc2c02b4 in this term.
I20250114 03:12:18.112111  4768 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 10.
I20250114 03:12:18.112289  4808 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c99007133884f7890f978a641b3a902, 6a4b32f73dce497a86df7c103c209107; no voters: 01b1d948fc9846fa99a040f4cc2c02b4
I20250114 03:12:18.112406  4974 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 10 FOLLOWER]: Leader election won for term 10
I20250114 03:12:18.112529  4974 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 10 LEADER]: Becoming Leader. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:18.112591  4974 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 9.22, Last appended by leader: 22, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.113205  4665 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 reported cstate change: term changed from 9 to 10, leader changed from 6a4b32f73dce497a86df7c103c209107 (127.4.10.129) to 2c99007133884f7890f978a641b3a902 (127.4.10.130). New cstate: current_term: 10 leader_uuid: "2c99007133884f7890f978a641b3a902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:18.145325  5015 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:18.145428  5015 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 10 FOLLOWER]: Advancing to term 11
I20250114 03:12:18.146214  5015 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.146389  5015 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 11 election: Requested vote from peers 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:18.146641  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 11 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:18.146704  4768 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 10 FOLLOWER]: Advancing to term 11
I20250114 03:12:18.146675  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" candidate_term: 11 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:18.147305  4768 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b1d948fc9846fa99a040f4cc2c02b4 in term 11.
I20250114 03:12:18.147480  4883 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 6a4b32f73dce497a86df7c103c209107; no voters: 2c99007133884f7890f978a641b3a902
I20250114 03:12:18.147578  5015 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 11 FOLLOWER]: Leader election won for term 11
I20250114 03:12:18.147636  5015 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 11 LEADER]: Becoming Leader. State: Replica: 01b1d948fc9846fa99a040f4cc2c02b4, State: Running, Role: LEADER
I20250114 03:12:18.147692  5015 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 9.22, Last appended by leader: 22, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.150789  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.151952  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.154083  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.158227  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.166445  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.182693  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.183120  4663 catalog_manager.cc:5526] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 reported cstate change: term changed from 10 to 11, leader changed from 2c99007133884f7890f978a641b3a902 (127.4.10.130) to 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131). New cstate: current_term: 11 leader_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:18.200124  4920 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 11 LEADER]: Rejecting Update request from peer 2c99007133884f7890f978a641b3a902 for earlier term 10. Current term is 11. Ops: []
I20250114 03:12:18.200465  4974 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 03:12:18.200603  4974 consensus_queue.cc:1225] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Peer 01b1d948fc9846fa99a040f4cc2c02b4 log is divergent from this leader: its last log entry 11.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20250114 03:12:18.200646  4974 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 10 LEADER]: Stepping down as leader of term 10
I20250114 03:12:18.200686  4974 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:18.200786  4974 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 10.23, Last appended by leader: 23, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.200886  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 10 FOLLOWER]: Advancing to term 11
I20250114 03:12:18.214974  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.225798  4768 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 11 FOLLOWER]: Refusing update from remote peer 01b1d948fc9846fa99a040f4cc2c02b4: Log matching property violated. Preceding OpId in replica: term: 9 index: 22. Preceding OpId from leader: term: 11 index: 23. (index mismatch)
I20250114 03:12:18.226097  5015 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 03:12:18.227085  4845 raft_consensus.cc:1270] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 11 FOLLOWER]: Refusing update from remote peer 01b1d948fc9846fa99a040f4cc2c02b4: Log matching property violated. Preceding OpId in replica: term: 10 index: 23. Preceding OpId from leader: term: 11 index: 23. (term mismatch)
I20250114 03:12:18.227140  4845 pending_rounds.cc:77] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Aborting all ops after (but not including) 22
I20250114 03:12:18.227162  4845 pending_rounds.cc:99] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Aborting uncommitted NO_OP operation due to leader change: 10.23
I20250114 03:12:18.227182  4845 raft_consensus.cc:2883] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 03:12:18.227363  4972 consensus_queue.cc:1035] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 03:12:18.279201  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.407447  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:18.407590  4138 log_reader.cc:385] T 69f16283f4bb432589185fa78ca4dbc9: removed 1 log segments from log reader
I20250114 03:12:18.407634  4138 log.cc:1078] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Deleting log segment in path: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root/wals/69f16283f4bb432589185fa78ca4dbc9/wal-000000001 (ops 1-18)
I20250114 03:12:18.407821  5005 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 01b1d948fc9846fa99a040f4cc2c02b4)
I20250114 03:12:18.407887  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 11 FOLLOWER]: Advancing to term 12
I20250114 03:12:18.408174  4138 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250114 03:12:18.408241  4138 master.cc:537] Master@127.4.10.190:34423 shutting down...
I20250114 03:12:18.408792  5005 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.409173  5005 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 12 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:18.409245  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 12 candidate_status { last_received { term: 11 index: 23 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:18.409324  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 12 candidate_status { last_received { term: 11 index: 23 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:18.409651  4732 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4b32f73dce497a86df7c103c209107; no voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902
I20250114 03:12:18.409780  5005 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250114 03:12:18.413295  4768 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 12 FOLLOWER]: Rejecting Update request from peer 01b1d948fc9846fa99a040f4cc2c02b4 for earlier term 11. Current term is 12. Ops: []
I20250114 03:12:18.413501  4972 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 }, Status: INVALID_TERM, Last received: 11.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250114 03:12:18.413609  4972 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 11 LEADER]: Stepping down as leader of term 11
I20250114 03:12:18.413650  4972 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 01b1d948fc9846fa99a040f4cc2c02b4, State: Running, Role: LEADER
I20250114 03:12:18.413715  4972 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 11.23, Last appended by leader: 23, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.413794  4972 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 11 FOLLOWER]: Advancing to term 12
I20250114 03:12:18.447461  4974 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 01b1d948fc9846fa99a040f4cc2c02b4)
I20250114 03:12:18.447577  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 11 FOLLOWER]: Advancing to term 12
I20250114 03:12:18.448210  4974 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.448364  4974 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 12 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253)
I20250114 03:12:18.448575  4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 12 candidate_status { last_received { term: 11 index: 23 } } ignore_live_leader: false dest_uuid: "6a4b32f73dce497a86df7c103c209107"
I20250114 03:12:18.448653  4768 raft_consensus.cc:2388] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c99007133884f7890f978a641b3a902 in current term 12: Already voted for candidate 6a4b32f73dce497a86df7c103c209107 in this term.
I20250114 03:12:18.448590  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "2c99007133884f7890f978a641b3a902" candidate_term: 12 candidate_status { last_received { term: 11 index: 23 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:18.449463  4920 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c99007133884f7890f978a641b3a902 in term 12.
I20250114 03:12:18.449667  4809 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902; no voters: 6a4b32f73dce497a86df7c103c209107
I20250114 03:12:18.449750  4974 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 12 FOLLOWER]: Leader election won for term 12
I20250114 03:12:18.449810  4974 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 12 LEADER]: Becoming Leader. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:18.449887  4974 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 11.23, Last appended by leader: 23, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
W20250114 03:12:18.450878  4871 heartbeater.cc:643] Failed to heartbeat to 127.4.10.190:34423 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20250114 03:12:18.525188  4138 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:12:18.525297  4138 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:18.525319  4138 tablet_replica.cc:331] T 00000000000000000000000000000000 P 062e89d1e061457681905c352f30f818: stopping tablet replica
I20250114 03:12:18.537569  4138 master.cc:559] Master@127.4.10.190:34423 shutdown complete.
I20250114 03:12:18.538844  4138 tablet_server.cc:178] TabletServer@127.4.10.129:0 shutting down...
I20250114 03:12:18.541393  5005 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:18.541471  5005 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 12 FOLLOWER]: Advancing to term 13
I20250114 03:12:18.542083  5005 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.542340  5005 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 13 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
I20250114 03:12:18.542418  4845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 13 candidate_status { last_received { term: 11 index: 23 } } ignore_live_leader: false dest_uuid: "2c99007133884f7890f978a641b3a902"
I20250114 03:12:18.542434  4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f16283f4bb432589185fa78ca4dbc9" candidate_uuid: "6a4b32f73dce497a86df7c103c209107" candidate_term: 13 candidate_status { last_received { term: 11 index: 23 } } ignore_live_leader: false dest_uuid: "01b1d948fc9846fa99a040f4cc2c02b4"
I20250114 03:12:18.542634  4920 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 12 FOLLOWER]: Advancing to term 13
W20250114 03:12:18.542949  4808 consensus_peers.cc:487] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 -> Peer 6a4b32f73dce497a86df7c103c209107 (127.4.10.129:41253): Couldn't send request to peer 6a4b32f73dce497a86df7c103c209107. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250114 03:12:18.543263  4920 raft_consensus.cc:2463] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4b32f73dce497a86df7c103c209107 in term 13.
I20250114 03:12:18.543431  4732 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01b1d948fc9846fa99a040f4cc2c02b4, 6a4b32f73dce497a86df7c103c209107; no voters: 2c99007133884f7890f978a641b3a902
I20250114 03:12:18.543515  5005 raft_consensus.cc:2798] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 FOLLOWER]: Leader election won for term 13
I20250114 03:12:18.543623  5005 raft_consensus.cc:695] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 LEADER]: Becoming Leader. State: Replica: 6a4b32f73dce497a86df7c103c209107, State: Running, Role: LEADER
I20250114 03:12:18.543682  5005 consensus_queue.cc:237] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 11.23, Last appended by leader: 23, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.556160  4920 raft_consensus.cc:1235] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 13 FOLLOWER]: Rejecting Update request from peer 2c99007133884f7890f978a641b3a902 for earlier term 12. Current term is 13. Ops: []
I20250114 03:12:18.556413  4974 consensus_queue.cc:1046] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250114 03:12:18.556512  4974 raft_consensus.cc:3049] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 12 LEADER]: Stepping down as leader of term 12
I20250114 03:12:18.556545  4974 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2c99007133884f7890f978a641b3a902, State: Running, Role: LEADER
I20250114 03:12:18.556610  4974 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 12.24, Last appended by leader: 24, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.556684  4974 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 12 FOLLOWER]: Advancing to term 13
I20250114 03:12:18.586458  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:18.586719  4138 tablet_replica.cc:331] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: stopping tablet replica
I20250114 03:12:18.586810  4138 raft_consensus.cc:2238] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 LEADER]: Raft consensus shutting down.
I20250114 03:12:18.586869  4138 pending_rounds.cc:62] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Trying to abort 1 pending ops.
I20250114 03:12:18.586889  4138 pending_rounds.cc:69] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Aborting op as it isn't in flight: id { term: 13 index: 24 } timestamp: 7114032490675363840 op_type: NO_OP noop_request { }
I20250114 03:12:18.586951  4138 raft_consensus.cc:2883] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:12:18.586977  4138 raft_consensus.cc:2267] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:18.598994  4138 tablet_server.cc:195] TabletServer@127.4.10.129:0 shutdown complete.
I20250114 03:12:18.599982  4138 tablet_server.cc:178] TabletServer@127.4.10.130:0 shutting down...
I20250114 03:12:18.613782  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:18.613968  4138 tablet_replica.cc:331] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: stopping tablet replica
I20250114 03:12:18.614042  4138 raft_consensus.cc:2238] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 13 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:18.614110  4138 pending_rounds.cc:62] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Trying to abort 1 pending ops.
I20250114 03:12:18.614140  4138 pending_rounds.cc:69] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902: Aborting op as it isn't in flight: id { term: 12 index: 24 } timestamp: 7114032490291228672 op_type: NO_OP noop_request { }
I20250114 03:12:18.614223  4138 raft_consensus.cc:2883] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:12:18.614262  4138 raft_consensus.cc:2267] T 69f16283f4bb432589185fa78ca4dbc9 P 2c99007133884f7890f978a641b3a902 [term 13 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:18.626379  4138 tablet_server.cc:195] TabletServer@127.4.10.130:0 shutdown complete.
I20250114 03:12:18.627350  4138 tablet_server.cc:178] TabletServer@127.4.10.131:0 shutting down...
I20250114 03:12:18.636933  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:18.637079  4138 tablet_replica.cc:331] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4: stopping tablet replica
I20250114 03:12:18.637135  4138 raft_consensus.cc:2238] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 13 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:18.637193  4138 raft_consensus.cc:2267] T 69f16283f4bb432589185fa78ca4dbc9 P 01b1d948fc9846fa99a040f4cc2c02b4 [term 13 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:18.638844  4138 tablet_server.cc:195] TabletServer@127.4.10.131:0 shutdown complete.
I20250114 03:12:18.639653  4138 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250114 03:12:18.639803  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:18.642024  5025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:18.642659  5027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:18.643494  5024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:18.644203  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:18.644316  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:18.644351  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:18.644373  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824338644373 us; error 0 us; skew 500 ppm
I20250114 03:12:18.653656  4138 webserver.cc:458] Webserver started at http://127.4.10.129:38617/ using document root <none> and password file <none>
I20250114 03:12:18.653788  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:18.653836  4138 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 03:12:18.654584  4138 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.000s
I20250114 03:12:18.656002  5035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:18.656183  4138 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:18.656239  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root
uuid: "6a4b32f73dce497a86df7c103c209107"
format_stamp: "Formatted at 2025-01-14 03:12:16 on dist-test-slave-ppl6"
I20250114 03:12:18.656299  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1736824303196632-4138-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5430
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:18.667138  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:18.667356  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:18.667516  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:18.667860  5042 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 03:12:18.668624  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 03:12:18.668673  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
I20250114 03:12:18.668721  4138 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 03:12:18.669138  4138 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 03:12:18.669167  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:18.669214  5042 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Bootstrap starting.
I20250114 03:12:18.670667  5042 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250114 03:12:18.670934  5042 tablet_bootstrap.cc:492] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Bootstrap complete.
I20250114 03:12:18.671028  5042 ts_tablet_manager.cc:1397] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20250114 03:12:18.671162  5042 raft_consensus.cc:357] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.671244  5042 raft_consensus.cc:738] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4b32f73dce497a86df7c103c209107, State: Initialized, Role: FOLLOWER
I20250114 03:12:18.671309  5042 consensus_queue.cc:260] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 13.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.671471  5042 ts_tablet_manager.cc:1428] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:12:18.672426  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.129:41253
I20250114 03:12:18.674872  5101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.129:41253 every 8 connection(s)
W20250114 03:12:18.675793  5102 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250114 03:12:18.681928  4138 tablet_server.cc:178] TabletServer@127.4.10.129:41253 shutting down...
I20250114 03:12:18.786269  5083 raft_consensus.cc:491] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:18.786378  5083 raft_consensus.cc:3054] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 13 FOLLOWER]: Advancing to term 14
I20250114 03:12:18.787127  5083 raft_consensus.cc:513] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b1d948fc9846fa99a040f4cc2c02b4" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 33927 } } peers { permanent_uuid: "6a4b32f73dce497a86df7c103c209107" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41253 } } peers { permanent_uuid: "2c99007133884f7890f978a641b3a902" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 38443 } }
I20250114 03:12:18.787315  5083 leader_election.cc:290] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 14 election: Requested vote from peers 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927), 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443)
W20250114 03:12:18.788118  5037 leader_election.cc:336] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 01b1d948fc9846fa99a040f4cc2c02b4 (127.4.10.131:33927): Network error: Client connection negotiation failed: client connection to 127.4.10.131:33927: connect: Connection refused (error 111)
W20250114 03:12:18.788208  5038 leader_election.cc:336] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 2c99007133884f7890f978a641b3a902 (127.4.10.130:38443): Network error: Client connection negotiation failed: client connection to 127.4.10.130:38443: connect: Connection refused (error 111)
I20250114 03:12:18.788262  5038 leader_election.cc:304] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4b32f73dce497a86df7c103c209107; no voters: 01b1d948fc9846fa99a040f4cc2c02b4, 2c99007133884f7890f978a641b3a902
I20250114 03:12:18.788337  5083 raft_consensus.cc:2743] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250114 03:12:18.925420  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:18.925668  4138 tablet_replica.cc:331] stopping tablet replica
I20250114 03:12:18.925750  4138 raft_consensus.cc:2238] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 14 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:18.925827  4138 pending_rounds.cc:62] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Trying to abort 1 pending ops.
I20250114 03:12:18.925875  4138 pending_rounds.cc:69] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107: Aborting op as it isn't in flight: id { term: 13 index: 24 } timestamp: 7114032490675363840 op_type: NO_OP noop_request { }
I20250114 03:12:18.925943  4138 raft_consensus.cc:2883] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:12:18.925964  4138 raft_consensus.cc:2267] T 69f16283f4bb432589185fa78ca4dbc9 P 6a4b32f73dce497a86df7c103c209107 [term 14 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:18.937907  4138 tablet_server.cc:195] TabletServer@127.4.10.129:41253 shutdown complete.
[       OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (2933 ms)
[ RUN      ] TimestampAdvancementITest.Kudu2463Test
I20250114 03:12:18.942852  4138 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.10.190:41851
I20250114 03:12:18.943018  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:18.945031  5109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:18.945653  5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:18.946280  5112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:18.947208  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:18.947355  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:18.947383  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:18.947396  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824338947396 us; error 0 us; skew 500 ppm
I20250114 03:12:18.948072  4138 webserver.cc:458] Webserver started at http://127.4.10.190:39319/ using document root <none> and password file <none>
I20250114 03:12:18.948143  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:18.948181  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:18.948217  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:18.948474  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/master-0-root/instance:
uuid: "d01395cc5c2d4a9b859201c4e3a9176f"
format_stamp: "Formatted at 2025-01-14 03:12:18 on dist-test-slave-ppl6"
I20250114 03:12:18.949375  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:18.950042  5117 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:18.950213  4138 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I20250114 03:12:18.950270  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/master-0-root
uuid: "d01395cc5c2d4a9b859201c4e3a9176f"
format_stamp: "Formatted at 2025-01-14 03:12:18 on dist-test-slave-ppl6"
I20250114 03:12:18.950313  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:18.958819  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:18.959028  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:18.961597  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.190:41851
I20250114 03:12:18.961730  5170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.190:41851 every 8 connection(s)
I20250114 03:12:18.962244  5171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:18.963248  5171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f: Bootstrap starting.
I20250114 03:12:18.963541  5171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:18.964038  5171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f: No bootstrap required, opened a new log
I20250114 03:12:18.964161  5171 raft_consensus.cc:357] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" member_type: VOTER }
I20250114 03:12:18.964210  5171 raft_consensus.cc:383] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:18.964223  5171 raft_consensus.cc:738] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01395cc5c2d4a9b859201c4e3a9176f, State: Initialized, Role: FOLLOWER
I20250114 03:12:18.964262  5171 consensus_queue.cc:260] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" member_type: VOTER }
I20250114 03:12:18.964321  5171 raft_consensus.cc:397] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:12:18.964351  5171 raft_consensus.cc:491] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:12:18.964366  5171 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:18.964869  5171 raft_consensus.cc:513] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" member_type: VOTER }
I20250114 03:12:18.964926  5171 leader_election.cc:304] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d01395cc5c2d4a9b859201c4e3a9176f; no voters: 
I20250114 03:12:18.965011  5171 leader_election.cc:290] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20250114 03:12:18.965081  5174 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:12:18.965153  5171 sys_catalog.cc:564] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:12:18.965214  5174 raft_consensus.cc:695] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 1 LEADER]: Becoming Leader. State: Replica: d01395cc5c2d4a9b859201c4e3a9176f, State: Running, Role: LEADER
I20250114 03:12:18.965266  5174 consensus_queue.cc:237] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" member_type: VOTER }
I20250114 03:12:18.965444  5174 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d01395cc5c2d4a9b859201c4e3a9176f. Latest consensus state: current_term: 1 leader_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" member_type: VOTER } }
I20250114 03:12:18.965477  5175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01395cc5c2d4a9b859201c4e3a9176f" member_type: VOTER } }
I20250114 03:12:18.965503  5174 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:18.965819  5175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:18.965981  5182 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:12:18.966109  5182 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:12:18.966538  4138 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 03:12:18.966660  5182 catalog_manager.cc:1348] Generated new cluster ID: c7981c9ec85c48849306b1f0427e7cf7
I20250114 03:12:18.966697  5182 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:12:18.986496  5182 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:12:18.986924  5182 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:12:18.996114  5182 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f: Generated new TSK 0
I20250114 03:12:18.996217  5182 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:12:18.998325  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:19.000522  5192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:19.001122  4138 server_base.cc:1034] running on GCE node
W20250114 03:12:19.001700  5193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:19.002593  5195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:19.002748  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:19.002816  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:19.002838  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824339002838 us; error 0 us; skew 500 ppm
I20250114 03:12:19.003702  4138 webserver.cc:458] Webserver started at http://127.4.10.129:41869/ using document root <none> and password file <none>
I20250114 03:12:19.003816  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:19.003883  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:19.003952  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:19.004285  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root/instance:
uuid: "20365930042c420696f9ebf89e533a1a"
format_stamp: "Formatted at 2025-01-14 03:12:19 on dist-test-slave-ppl6"
I20250114 03:12:19.005349  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:19.005947  5200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.006096  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:12:19.006147  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root
uuid: "20365930042c420696f9ebf89e533a1a"
format_stamp: "Formatted at 2025-01-14 03:12:19 on dist-test-slave-ppl6"
I20250114 03:12:19.006199  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:19.012506  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:19.012753  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:19.012916  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:19.013121  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:19.013149  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.013183  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:19.013211  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.015986  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.129:37323
I20250114 03:12:19.016022  5264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.129:37323 every 8 connection(s)
I20250114 03:12:19.016371  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:19.018461  5269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:19.019117  5272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:19.020067  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:19.020170  5265 heartbeater.cc:346] Connected to a master server at 127.4.10.190:41851
I20250114 03:12:19.020224  5265 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:19.020309  5265 heartbeater.cc:510] Master 127.4.10.190:41851 requested a full tablet report, sending...
I20250114 03:12:19.020540  5134 ts_manager.cc:194] Registered new tserver with Master: 20365930042c420696f9ebf89e533a1a (127.4.10.129:37323)
W20250114 03:12:19.020813  5270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:19.020990  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:19.021031  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:19.021045  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824339021045 us; error 0 us; skew 500 ppm
I20250114 03:12:19.021149  5134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50122
I20250114 03:12:19.022047  4138 webserver.cc:458] Webserver started at http://127.4.10.130:36241/ using document root <none> and password file <none>
I20250114 03:12:19.022142  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:19.022200  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:19.022259  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:19.022600  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-1-root/instance:
uuid: "9e38629a4fd8409da258e194edc2bed5"
format_stamp: "Formatted at 2025-01-14 03:12:19 on dist-test-slave-ppl6"
I20250114 03:12:19.023626  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:19.024194  5277 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.024348  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.001s
I20250114 03:12:19.024401  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-1-root
uuid: "9e38629a4fd8409da258e194edc2bed5"
format_stamp: "Formatted at 2025-01-14 03:12:19 on dist-test-slave-ppl6"
I20250114 03:12:19.024456  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:19.031442  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:19.031716  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:19.031875  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:19.032099  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:19.032130  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.032156  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:19.032171  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.035619  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.130:45207
I20250114 03:12:19.035660  5340 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.130:45207 every 8 connection(s)
I20250114 03:12:19.036065  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:19.038156  5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:19.038735  5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:19.039129  5341 heartbeater.cc:346] Connected to a master server at 127.4.10.190:41851
I20250114 03:12:19.039192  5341 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:19.039305  5341 heartbeater.cc:510] Master 127.4.10.190:41851 requested a full tablet report, sending...
W20250114 03:12:19.039366  5347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:19.039546  5134 ts_manager.cc:194] Registered new tserver with Master: 9e38629a4fd8409da258e194edc2bed5 (127.4.10.130:45207)
I20250114 03:12:19.039908  5134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50134
I20250114 03:12:19.040284  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:19.040395  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:19.040417  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:19.040431  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824339040430 us; error 0 us; skew 500 ppm
I20250114 03:12:19.041071  4138 webserver.cc:458] Webserver started at http://127.4.10.131:40581/ using document root <none> and password file <none>
I20250114 03:12:19.041142  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:19.041181  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:19.041216  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:19.041476  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-2-root/instance:
uuid: "c31b71af4b7c4ed2bd7f243a6c89130b"
format_stamp: "Formatted at 2025-01-14 03:12:19 on dist-test-slave-ppl6"
I20250114 03:12:19.042418  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:19.042872  5352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.043007  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.043056  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-2-root
uuid: "c31b71af4b7c4ed2bd7f243a6c89130b"
format_stamp: "Formatted at 2025-01-14 03:12:19 on dist-test-slave-ppl6"
I20250114 03:12:19.043102  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:19.052826  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:19.053061  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:19.053202  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:19.053412  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:19.053442  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.053464  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:19.053494  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.056747  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.131:38573
I20250114 03:12:19.056828  5416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.131:38573 every 8 connection(s)
I20250114 03:12:19.059976  5417 heartbeater.cc:346] Connected to a master server at 127.4.10.190:41851
I20250114 03:12:19.060046  5417 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:19.060158  5417 heartbeater.cc:510] Master 127.4.10.190:41851 requested a full tablet report, sending...
I20250114 03:12:19.060428  5134 ts_manager.cc:194] Registered new tserver with Master: c31b71af4b7c4ed2bd7f243a6c89130b (127.4.10.131:38573)
I20250114 03:12:19.060777  5134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50146
I20250114 03:12:19.061393  4138 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004438664s
I20250114 03:12:19.069022  4138 test_util.cc:274] Using random seed: -305891236
I20250114 03:12:19.074085  5134 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50156:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20250114 03:12:19.074225  5134 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:12:19.078743  5305 tablet_service.cc:1467] Processing CreateTablet for tablet dfec1481076e4b6e9daa39e8205047ec (DEFAULT_TABLE table=test-workload [id=40fe16353f404b8cad830efa4edd1317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:12:19.078915  5305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfec1481076e4b6e9daa39e8205047ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:19.079301  5228 tablet_service.cc:1467] Processing CreateTablet for tablet dfec1481076e4b6e9daa39e8205047ec (DEFAULT_TABLE table=test-workload [id=40fe16353f404b8cad830efa4edd1317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:12:19.079371  5381 tablet_service.cc:1467] Processing CreateTablet for tablet dfec1481076e4b6e9daa39e8205047ec (DEFAULT_TABLE table=test-workload [id=40fe16353f404b8cad830efa4edd1317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:12:19.079419  5228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfec1481076e4b6e9daa39e8205047ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:19.079488  5381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfec1481076e4b6e9daa39e8205047ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:19.080504  5438 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5: Bootstrap starting.
I20250114 03:12:19.080996  5440 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b: Bootstrap starting.
I20250114 03:12:19.081095  5439 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Bootstrap starting.
I20250114 03:12:19.080997  5438 tablet_bootstrap.cc:654] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:19.081568  5440 tablet_bootstrap.cc:654] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:19.081573  5439 tablet_bootstrap.cc:654] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:19.081761  5438 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5: No bootstrap required, opened a new log
I20250114 03:12:19.081809  5438 ts_tablet_manager.cc:1397] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:19.082017  5438 raft_consensus.cc:357] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.082110  5438 raft_consensus.cc:383] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:19.082139  5438 raft_consensus.cc:738] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e38629a4fd8409da258e194edc2bed5, State: Initialized, Role: FOLLOWER
I20250114 03:12:19.082197  5438 consensus_queue.cc:260] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.082242  5440 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b: No bootstrap required, opened a new log
I20250114 03:12:19.082298  5440 ts_tablet_manager.cc:1397] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:19.082348  5438 ts_tablet_manager.cc:1428] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.082429  5341 heartbeater.cc:502] Master 127.4.10.190:41851 was elected leader, sending a full tablet report...
I20250114 03:12:19.082454  5440 raft_consensus.cc:357] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.082499  5439 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: No bootstrap required, opened a new log
I20250114 03:12:19.082540  5440 raft_consensus.cc:383] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:19.082561  5440 raft_consensus.cc:738] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31b71af4b7c4ed2bd7f243a6c89130b, State: Initialized, Role: FOLLOWER
I20250114 03:12:19.082564  5439 ts_tablet_manager.cc:1397] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:19.082645  5440 consensus_queue.cc:260] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.082760  5439 raft_consensus.cc:357] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.082924  5439 raft_consensus.cc:383] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:19.082969  5439 raft_consensus.cc:738] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20365930042c420696f9ebf89e533a1a, State: Initialized, Role: FOLLOWER
I20250114 03:12:19.082814  5440 ts_tablet_manager.cc:1428] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:12:19.082818  5417 heartbeater.cc:502] Master 127.4.10.190:41851 was elected leader, sending a full tablet report...
I20250114 03:12:19.083021  5439 consensus_queue.cc:260] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.083176  5439 ts_tablet_manager.cc:1428] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20250114 03:12:19.083249  5265 heartbeater.cc:502] Master 127.4.10.190:41851 was elected leader, sending a full tablet report...
I20250114 03:12:19.086359  5444 raft_consensus.cc:491] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:12:19.086417  5444 raft_consensus.cc:513] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.086589  5444 leader_election.cc:290] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c31b71af4b7c4ed2bd7f243a6c89130b (127.4.10.131:38573), 20365930042c420696f9ebf89e533a1a (127.4.10.129:37323)
I20250114 03:12:19.089047  5391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfec1481076e4b6e9daa39e8205047ec" candidate_uuid: "9e38629a4fd8409da258e194edc2bed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" is_pre_election: true
I20250114 03:12:19.089138  5391 raft_consensus.cc:2463] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e38629a4fd8409da258e194edc2bed5 in term 0.
I20250114 03:12:19.089160  5239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfec1481076e4b6e9daa39e8205047ec" candidate_uuid: "9e38629a4fd8409da258e194edc2bed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20365930042c420696f9ebf89e533a1a" is_pre_election: true
I20250114 03:12:19.089243  5239 raft_consensus.cc:2463] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e38629a4fd8409da258e194edc2bed5 in term 0.
I20250114 03:12:19.089280  5281 leader_election.cc:304] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e38629a4fd8409da258e194edc2bed5, c31b71af4b7c4ed2bd7f243a6c89130b; no voters: 
I20250114 03:12:19.089377  5444 raft_consensus.cc:2798] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:12:19.089427  5444 raft_consensus.cc:491] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:12:19.089447  5444 raft_consensus.cc:3054] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:19.090129  5444 raft_consensus.cc:513] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.090246  5444 leader_election.cc:290] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [CANDIDATE]: Term 1 election: Requested vote from peers c31b71af4b7c4ed2bd7f243a6c89130b (127.4.10.131:38573), 20365930042c420696f9ebf89e533a1a (127.4.10.129:37323)
I20250114 03:12:19.090365  5239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfec1481076e4b6e9daa39e8205047ec" candidate_uuid: "9e38629a4fd8409da258e194edc2bed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20365930042c420696f9ebf89e533a1a"
I20250114 03:12:19.090410  5239 raft_consensus.cc:3054] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:19.091158  5239 raft_consensus.cc:2463] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e38629a4fd8409da258e194edc2bed5 in term 1.
I20250114 03:12:19.091259  5391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfec1481076e4b6e9daa39e8205047ec" candidate_uuid: "9e38629a4fd8409da258e194edc2bed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b"
I20250114 03:12:19.091311  5391 raft_consensus.cc:3054] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:19.091394  5279 leader_election.cc:304] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20365930042c420696f9ebf89e533a1a, 9e38629a4fd8409da258e194edc2bed5; no voters: 
I20250114 03:12:19.091960  5391 raft_consensus.cc:2463] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e38629a4fd8409da258e194edc2bed5 in term 1.
I20250114 03:12:19.092046  5444 raft_consensus.cc:2798] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:12:19.092167  5444 raft_consensus.cc:695] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Becoming Leader. State: Replica: 9e38629a4fd8409da258e194edc2bed5, State: Running, Role: LEADER
I20250114 03:12:19.092228  5444 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.092816  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e38629a4fd8409da258e194edc2bed5 (127.4.10.130). New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:19.101344  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:12:19.101344  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:12:19.101648  5448 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:12:19.101754  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:12:19.122015  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.122469  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250114 03:12:19.122488  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250114 03:12:19.122704  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 03:12:19.122826  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250114 03:12:19.123459  5448 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.123586  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.123567  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.124715  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index -1 to 17, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.125296  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.125725  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
W20250114 03:12:19.125842  5133 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 17: no extra replica candidate found for tablet dfec1481076e4b6e9daa39e8205047ec (table test-workload [id=40fe16353f404b8cad830efa4edd1317]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250114 03:12:19.125970  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 03:12:19.126157  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250114 03:12:19.126338  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250114 03:12:19.126598  5472 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.126857  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.127086  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.127902  5133 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a reported cstate change: config changed from index 17 to 18, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.128161  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.128618  5390 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250114 03:12:19.128767  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250114 03:12:19.128960  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 03:12:19.129326  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250114 03:12:19.129810  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.129949  5390 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.130877  5133 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index 18 to 19, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.130888  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.130972  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.131295  5390 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250114 03:12:19.131458  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 03:12:19.132066  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250114 03:12:19.132138  5448 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.132263  5390 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.133060  5448 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250114 03:12:19.133172  5131 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index 19 to 20, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.133334  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.133769  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.134163  5390 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250114 03:12:19.134327  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 03:12:19.134513  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250114 03:12:19.134696  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250114 03:12:19.134900  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.134996  5390 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.135280  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.135871  5133 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index 20 to 21, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.136006  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.136333  5390 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250114 03:12:19.136508  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250114 03:12:19.136732  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250114 03:12:19.136899  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250114 03:12:19.137012  5472 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.137163  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.137167  5390 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.138015  5133 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a reported cstate change: config changed from index 21 to 22, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.138463  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.138783  5390 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250114 03:12:19.138860  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250114 03:12:19.138960  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 03:12:19.139120  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250114 03:12:19.139452  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.139581  5390 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.140362  5131 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: config changed from index 22 to 23, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:19.140496  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.140792  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.141115  5390 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250114 03:12:19.141372  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250114 03:12:19.141551  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250114 03:12:19.141718  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250114 03:12:19.142084  5448 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.142207  5390 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.142940  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: config changed from index 23 to 24, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:19.143021  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.143065  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.143397  5390 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250114 03:12:19.143517  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250114 03:12:19.143724  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250114 03:12:19.143927  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250114 03:12:19.144296  5448 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.144681  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.145234  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: config changed from index 24 to 25, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:19.145264  5390 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.145510  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.145920  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250114 03:12:19.146101  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250114 03:12:19.146178  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250114 03:12:19.146392  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250114 03:12:19.146776  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.146946  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.147106  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.148017  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index 25 to 26, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.148239  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.148609  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250114 03:12:19.148777  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250114 03:12:19.148763  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250114 03:12:19.148975  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250114 03:12:19.149269  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.149367  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.150208  5131 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: config changed from index 26 to 27, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: UNKNOWN } } }
I20250114 03:12:19.150278  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.150278  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.151046  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250114 03:12:19.151278  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250114 03:12:19.151347  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250114 03:12:19.151517  5448 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250114 03:12:19.152006  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.152082  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.152169  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.153059  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: config changed from index 27 to 28, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:19.153154  5119 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 17: aborting the task: latest config opid_index 27; task opid_index 17
I20250114 03:12:19.153642  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.153998  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250114 03:12:19.154052  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250114 03:12:19.154207  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250114 03:12:19.154302  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250114 03:12:19.154816  5472 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.154923  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.154939  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.155903  5133 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index 28 to 29, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.156102  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.156463  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250114 03:12:19.156698  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250114 03:12:19.156672  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250114 03:12:19.156860  5448 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250114 03:12:19.157325  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.157428  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.158154  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.158213  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index 29 to 30, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.158360  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.158824  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250114 03:12:19.159013  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250114 03:12:19.159229  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250114 03:12:19.159534  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250114 03:12:19.159726  5448 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.160023  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.160344  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.160862  5131 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: config changed from index 30 to 31, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:19.161280  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.161825  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250114 03:12:19.162011  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250114 03:12:19.162021  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250114 03:12:19.162264  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250114 03:12:19.162590  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.162695  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.162931  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.163595  5133 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a reported cstate change: config changed from index 31 to 32, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.164000  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.164451  5239 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250114 03:12:19.164462  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250114 03:12:19.164623  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250114 03:12:19.164716  5448 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250114 03:12:19.165194  5448 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.165321  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.166141  5131 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b reported cstate change: config changed from index 32 to 33, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.166399  5239 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.166414  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.166890  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250114 03:12:19.167188  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250114 03:12:19.167285  5448 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250114 03:12:19.167363  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250114 03:12:19.167817  5448 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.167955  5238 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.167912  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.168820  5131 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a reported cstate change: config changed from index 33 to 34, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.169610  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.169950  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250114 03:12:19.169970  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250114 03:12:19.170147  5472 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250114 03:12:19.170236  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250114 03:12:19.170778  5121 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 21: aborting the task: latest config opid_index 34; task opid_index 21
I20250114 03:12:19.170833  5472 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.170979  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.171648  5238 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.171746  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 reported cstate change: config changed from index 34 to 35, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: NON_VOTER last_known_addr { host: "127.4.10.129" port: 37323 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:19.172567  5315 consensus_queue.cc:237] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.172984  5238 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 LEARNER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250114 03:12:19.172984  5391 raft_consensus.cc:1270] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Refusing update from remote peer 9e38629a4fd8409da258e194edc2bed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250114 03:12:19.173177  5444 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250114 03:12:19.173239  5448 consensus_queue.cc:1035] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250114 03:12:19.173748  5444 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.173902  5238 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.174769  5134 catalog_manager.cc:5526] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a reported cstate change: config changed from index 35 to 36, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e38629a4fd8409da258e194edc2bed5" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.174820  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.174777  5391 raft_consensus.cc:2949] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 20365930042c420696f9ebf89e533a1a (127.4.10.129) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } } }
I20250114 03:12:19.176026  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.178194  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.182345  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.186554  5118 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 27: aborting the task: latest config opid_index 36; task opid_index 27
I20250114 03:12:19.186664  5118 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 25: aborting the task: latest config opid_index 36; task opid_index 25
I20250114 03:12:19.189957  5120 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 19: aborting the task: latest config opid_index 36; task opid_index 19
I20250114 03:12:19.190516  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.197137  5121 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 23: aborting the task: latest config opid_index 36; task opid_index 23
I20250114 03:12:19.199558  5119 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 31: aborting the task: latest config opid_index 36; task opid_index 31
I20250114 03:12:19.200374  5121 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 33: aborting the task: latest config opid_index 36; task opid_index 33
I20250114 03:12:19.206710  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.209789  5119 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 29: aborting the task: latest config opid_index 36; task opid_index 29
I20250114 03:12:19.212070  5119 catalog_manager.cc:5073] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfec1481076e4b6e9daa39e8205047ec with cas_config_opid_index 35: aborting the task: latest config opid_index 36; task opid_index 35
I20250114 03:12:19.239012  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.303308  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.431607  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.687891  4138 timestamp_advancement-itest.cc:261] GCing logs...
I20250114 03:12:19.688026  4138 log_reader.cc:385] T dfec1481076e4b6e9daa39e8205047ec: removed 1 log segments from log reader
I20250114 03:12:19.688061  4138 log.cc:1078] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Deleting log segment in path: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root/wals/dfec1481076e4b6e9daa39e8205047ec/wal-000000001 (ops 1-16)
I20250114 03:12:19.688752  4138 timestamp_advancement-itest.cc:268] Shutting down cluster...
I20250114 03:12:19.688832  4138 master.cc:537] Master@127.4.10.190:41851 shutting down...
I20250114 03:12:19.710076  4138 raft_consensus.cc:2238] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:12:19.710186  4138 raft_consensus.cc:2267] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:19.710207  4138 tablet_replica.cc:331] T 00000000000000000000000000000000 P d01395cc5c2d4a9b859201c4e3a9176f: stopping tablet replica
I20250114 03:12:19.722298  4138 master.cc:559] Master@127.4.10.190:41851 shutdown complete.
I20250114 03:12:19.723548  4138 tablet_server.cc:178] TabletServer@127.4.10.129:0 shutting down...
I20250114 03:12:19.767072  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:19.767266  4138 tablet_replica.cc:331] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: stopping tablet replica
I20250114 03:12:19.767354  4138 raft_consensus.cc:2238] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:19.767445  4138 raft_consensus.cc:2267] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:19.779310  4138 tablet_server.cc:195] TabletServer@127.4.10.129:0 shutdown complete.
I20250114 03:12:19.780508  4138 tablet_server.cc:178] TabletServer@127.4.10.130:0 shutting down...
I20250114 03:12:19.786443  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:19.786604  4138 tablet_replica.cc:331] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5: stopping tablet replica
I20250114 03:12:19.786679  4138 raft_consensus.cc:2238] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:12:19.786761  4138 raft_consensus.cc:2267] T dfec1481076e4b6e9daa39e8205047ec P 9e38629a4fd8409da258e194edc2bed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:19.798619  4138 tablet_server.cc:195] TabletServer@127.4.10.130:0 shutdown complete.
I20250114 03:12:19.799669  4138 tablet_server.cc:178] TabletServer@127.4.10.131:0 shutting down...
I20250114 03:12:19.807682  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:19.807893  4138 tablet_replica.cc:331] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b: stopping tablet replica
I20250114 03:12:19.807977  4138 raft_consensus.cc:2238] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:19.808197  4138 raft_consensus.cc:2267] T dfec1481076e4b6e9daa39e8205047ec P c31b71af4b7c4ed2bd7f243a6c89130b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:19.819772  4138 tablet_server.cc:195] TabletServer@127.4.10.131:0 shutdown complete.
I20250114 03:12:19.820712  4138 timestamp_advancement-itest.cc:276] Restarting single tablet server...
I20250114 03:12:19.820909  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:19.822921  5473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:19.823729  5476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:19.824316  5474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:19.824959  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:19.825069  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:19.825105  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:19.825134  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824339825133 us; error 0 us; skew 500 ppm
I20250114 03:12:19.825882  4138 webserver.cc:458] Webserver started at http://127.4.10.129:41869/ using document root <none> and password file <none>
I20250114 03:12:19.825979  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:19.826017  4138 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 03:12:19.826699  4138 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:12:19.828478  5483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.828652  4138 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20250114 03:12:19.828729  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root
uuid: "20365930042c420696f9ebf89e533a1a"
format_stamp: "Formatted at 2025-01-14 03:12:19 on dist-test-slave-ppl6"
I20250114 03:12:19.828787  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1736824303196632-4138-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5293
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:19.849620  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:19.849836  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:19.850039  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:19.850363  5490 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 03:12:19.850931  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 03:12:19.850970  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
I20250114 03:12:19.851006  4138 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 03:12:19.851424  4138 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 03:12:19.851454  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:19.851477  5490 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Bootstrap starting.
I20250114 03:12:19.853354  5490 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 03:12:19.853582  5490 tablet_bootstrap.cc:492] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Bootstrap complete.
I20250114 03:12:19.853681  5490 ts_tablet_manager.cc:1397] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20250114 03:12:19.853808  5490 raft_consensus.cc:357] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.853904  5490 raft_consensus.cc:738] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20365930042c420696f9ebf89e533a1a, State: Initialized, Role: FOLLOWER
I20250114 03:12:19.853960  5490 consensus_queue.cc:260] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c31b71af4b7c4ed2bd7f243a6c89130b" member_type: VOTER last_known_addr { host: "127.4.10.131" port: 38573 } } peers { permanent_uuid: "9e38629a4fd8409da258e194edc2bed5" member_type: VOTER last_known_addr { host: "127.4.10.130" port: 45207 } } peers { permanent_uuid: "20365930042c420696f9ebf89e533a1a" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 37323 } }
I20250114 03:12:19.854120  5490 ts_tablet_manager.cc:1428] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a: Time spent starting tablet: real 0.000s	user 0.002s	sys 0.000s
I20250114 03:12:19.854792  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.129:37323
I20250114 03:12:19.857318  5550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.129:37323 every 8 connection(s)
W20250114 03:12:19.858119  5551 heartbeater.cc:414] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250114 03:12:19.862939  5504 tablet_service.cc:2908] Rejecting scan request for tablet dfec1481076e4b6e9daa39e8205047ec: Uninitialized: clean time has not yet been initialized
I20250114 03:12:19.863183  4138 timestamp_advancement-itest.cc:441] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20250114 03:12:19.863298  4138 tablet_server.cc:178] TabletServer@127.4.10.129:37323 shutting down...
I20250114 03:12:20.107933  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:20.108189  4138 tablet_replica.cc:331] stopping tablet replica
I20250114 03:12:20.108266  4138 raft_consensus.cc:2238] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 03:12:20.108325  4138 raft_consensus.cc:2267] T dfec1481076e4b6e9daa39e8205047ec P 20365930042c420696f9ebf89e533a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:20.119848  4138 tablet_server.cc:195] TabletServer@127.4.10.129:37323 shutdown complete.
[       OK ] TimestampAdvancementITest.Kudu2463Test (1181 ms)
[ RUN      ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20250114 03:12:20.124555  4138 timestamp_advancement-itest.cc:454] Running with 1 replicas
I20250114 03:12:20.124733  4138 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.10.190:41269
I20250114 03:12:20.124884  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:20.126932  5557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.127664  5556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.128295  4138 server_base.cc:1034] running on GCE node
W20250114 03:12:20.128849  5559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.129004  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:20.129040  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:20.129053  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824340129053 us; error 0 us; skew 500 ppm
I20250114 03:12:20.129726  4138 webserver.cc:458] Webserver started at http://127.4.10.190:42573/ using document root <none> and password file <none>
I20250114 03:12:20.129828  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:20.129905  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:20.129941  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:20.130262  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root/instance:
uuid: "0c8660b5854e430d9d2cad8e8cd6ea54"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.131223  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:20.131685  5564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.131824  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.131868  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root
uuid: "0c8660b5854e430d9d2cad8e8cd6ea54"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.131922  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:20.139588  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:20.139838  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:20.142433  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.190:41269
I20250114 03:12:20.143023  5616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.190:41269 every 8 connection(s)
I20250114 03:12:20.143214  5617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:20.144243  5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: Bootstrap starting.
I20250114 03:12:20.144558  5617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:20.145030  5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: No bootstrap required, opened a new log
I20250114 03:12:20.145148  5617 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.145200  5617 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:20.145222  5617 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8660b5854e430d9d2cad8e8cd6ea54, State: Initialized, Role: FOLLOWER
I20250114 03:12:20.145283  5617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.145318  5617 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:12:20.145340  5617 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:12:20.145370  5617 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:20.145852  5617 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.145961  5617 leader_election.cc:304] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8660b5854e430d9d2cad8e8cd6ea54; no voters: 
I20250114 03:12:20.146062  5617 leader_election.cc:290] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20250114 03:12:20.146111  5620 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:12:20.146203  5617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:12:20.146283  5620 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8660b5854e430d9d2cad8e8cd6ea54, State: Running, Role: LEADER
I20250114 03:12:20.146344  5620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.146518  5620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8660b5854e430d9d2cad8e8cd6ea54. Latest consensus state: current_term: 1 leader_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER } }
I20250114 03:12:20.146618  5620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:20.146559  5621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER } }
I20250114 03:12:20.146759  5621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:20.146915  5627 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:12:20.147053  5627 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:12:20.147554  4138 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 03:12:20.147627  5627 catalog_manager.cc:1348] Generated new cluster ID: 42029c62df7e483889ebdab83dcc4c12
I20250114 03:12:20.147661  5627 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:12:20.158808  5627 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:12:20.159246  5627 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:12:20.161815  5627 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: Generated new TSK 0
I20250114 03:12:20.161939  5627 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:12:20.163234  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:20.165318  5638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.166023  4138 server_base.cc:1034] running on GCE node
W20250114 03:12:20.166577  5641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.167155  5639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.167273  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:20.167305  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:20.167317  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824340167318 us; error 0 us; skew 500 ppm
I20250114 03:12:20.167992  4138 webserver.cc:458] Webserver started at http://127.4.10.129:45123/ using document root <none> and password file <none>
I20250114 03:12:20.168073  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:20.168112  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:20.168146  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:20.168417  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root/instance:
uuid: "9c2b40255a11408fb99e70287305dfab"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.169301  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:20.169786  5646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.169935  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.169979  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root
uuid: "9c2b40255a11408fb99e70287305dfab"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.170032  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:20.175364  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:20.175586  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:20.175725  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:20.175920  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:20.175946  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.175979  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:20.176005  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.178797  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.129:41015
I20250114 03:12:20.178833  5710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.129:41015 every 8 connection(s)
I20250114 03:12:20.182216  5711 heartbeater.cc:346] Connected to a master server at 127.4.10.190:41269
I20250114 03:12:20.182289  5711 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:20.182404  5711 heartbeater.cc:510] Master 127.4.10.190:41269 requested a full tablet report, sending...
I20250114 03:12:20.182636  5581 ts_manager.cc:194] Registered new tserver with Master: 9c2b40255a11408fb99e70287305dfab (127.4.10.129:41015)
I20250114 03:12:20.183167  5581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38214
I20250114 03:12:20.183367  4138 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004393977s
I20250114 03:12:20.190827  4138 test_util.cc:274] Using random seed: -304769430
I20250114 03:12:20.195061  5581 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38244:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250114 03:12:20.198834  5674 tablet_service.cc:1467] Processing CreateTablet for tablet 669fbe4edad14c21a08237148db6c835 (DEFAULT_TABLE table=test-workload [id=994d459b57c5465593f8b06390435396]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:12:20.198964  5674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669fbe4edad14c21a08237148db6c835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:12:20.200269  5729 tablet_bootstrap.cc:492] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Bootstrap starting.
I20250114 03:12:20.200580  5729 tablet_bootstrap.cc:654] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Neither blocks nor log segments found. Creating new log.
I20250114 03:12:20.201036  5729 tablet_bootstrap.cc:492] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: No bootstrap required, opened a new log
I20250114 03:12:20.201078  5729 ts_tablet_manager.cc:1397] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:20.201224  5729 raft_consensus.cc:357] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.201272  5729 raft_consensus.cc:383] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:12:20.201296  5729 raft_consensus.cc:738] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2b40255a11408fb99e70287305dfab, State: Initialized, Role: FOLLOWER
I20250114 03:12:20.201349  5729 consensus_queue.cc:260] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.201395  5729 raft_consensus.cc:397] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:12:20.201421  5729 raft_consensus.cc:491] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:12:20.201452  5729 raft_consensus.cc:3054] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:12:20.201987  5729 raft_consensus.cc:513] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.202051  5729 leader_election.cc:304] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2b40255a11408fb99e70287305dfab; no voters: 
I20250114 03:12:20.202174  5729 leader_election.cc:290] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [CANDIDATE]: Term 1 election: Requested vote from peers 
I20250114 03:12:20.202230  5731 raft_consensus.cc:2798] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:12:20.202316  5729 ts_tablet_manager.cc:1428] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:20.202365  5731 raft_consensus.cc:695] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 LEADER]: Becoming Leader. State: Replica: 9c2b40255a11408fb99e70287305dfab, State: Running, Role: LEADER
I20250114 03:12:20.202423  5711 heartbeater.cc:502] Master 127.4.10.190:41269 was elected leader, sending a full tablet report...
I20250114 03:12:20.202418  5731 consensus_queue.cc:237] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.202931  5581 catalog_manager.cc:5526] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c2b40255a11408fb99e70287305dfab (127.4.10.129). New cstate: current_term: 1 leader_uuid: "9c2b40255a11408fb99e70287305dfab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } health_report { overall_health: HEALTHY } } }
I20250114 03:12:20.205241  4138 tablet_server.cc:178] TabletServer@127.4.10.129:0 shutting down...
I20250114 03:12:20.207696  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:20.207851  4138 tablet_replica.cc:331] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: stopping tablet replica
I20250114 03:12:20.207924  4138 raft_consensus.cc:2238] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:12:20.207964  4138 raft_consensus.cc:2267] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:20.219774  4138 tablet_server.cc:195] TabletServer@127.4.10.129:0 shutdown complete.
I20250114 03:12:20.220849  4138 master.cc:537] Master@127.4.10.190:41269 shutting down...
I20250114 03:12:20.223410  4138 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:12:20.223495  4138 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:20.223588  4138 tablet_replica.cc:331] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: stopping tablet replica
I20250114 03:12:20.235707  4138 master.cc:559] Master@127.4.10.190:41269 shutdown complete.
I20250114 03:12:20.237355  4138 timestamp_advancement-itest.cc:454] Running with 3 replicas
I20250114 03:12:20.237594  4138 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.10.190:45057
I20250114 03:12:20.237717  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:20.239738  5739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.240406  5737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.241021  5736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.241632  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:20.241719  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:20.241748  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:20.241770  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824340241770 us; error 0 us; skew 500 ppm
I20250114 03:12:20.242486  4138 webserver.cc:458] Webserver started at http://127.4.10.190:34339/ using document root <none> and password file <none>
I20250114 03:12:20.242569  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:20.242614  4138 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 03:12:20.243187  4138 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.001s	sys 0.000s
I20250114 03:12:20.243732  5744 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.243885  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.243939  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root
uuid: "0c8660b5854e430d9d2cad8e8cd6ea54"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.244000  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:20.249182  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:20.249359  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:20.251928  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.190:45057
I20250114 03:12:20.252861  5797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.190:45057 every 8 connection(s)
I20250114 03:12:20.253425  5798 sys_catalog.cc:263] Verifying existing consensus state
I20250114 03:12:20.253805  5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: Bootstrap starting.
I20250114 03:12:20.255390  5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 03:12:20.255627  5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: Bootstrap complete.
I20250114 03:12:20.255828  5798 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.255887  5798 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8660b5854e430d9d2cad8e8cd6ea54, State: Initialized, Role: FOLLOWER
I20250114 03:12:20.255929  5798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.255972  5798 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:12:20.255992  5798 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:12:20.256016  5798 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:12:20.256480  5798 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.256531  5798 leader_election.cc:304] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8660b5854e430d9d2cad8e8cd6ea54; no voters: 
I20250114 03:12:20.256628  5798 leader_election.cc:290] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [CANDIDATE]: Term 2 election: Requested vote from peers 
I20250114 03:12:20.256686  5801 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:12:20.256760  5798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:12:20.256866  5801 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 2 LEADER]: Becoming Leader. State: Replica: 0c8660b5854e430d9d2cad8e8cd6ea54, State: Running, Role: LEADER
I20250114 03:12:20.256925  5801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER }
I20250114 03:12:20.257135  5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER } }
I20250114 03:12:20.257166  5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8660b5854e430d9d2cad8e8cd6ea54. Latest consensus state: current_term: 2 leader_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8660b5854e430d9d2cad8e8cd6ea54" member_type: VOTER } }
I20250114 03:12:20.257290  5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:20.257277  5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [sys.catalog]: This master's current role is: LEADER
I20250114 03:12:20.257448  5809 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:12:20.257754  5809 catalog_manager.cc:670] Loaded metadata for table test-workload [id=994d459b57c5465593f8b06390435396]
I20250114 03:12:20.257959  5809 tablet_loader.cc:96] loaded metadata for tablet 669fbe4edad14c21a08237148db6c835 (table test-workload [id=994d459b57c5465593f8b06390435396])
I20250114 03:12:20.258018  5809 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:12:20.258095  5809 catalog_manager.cc:1260] Loaded cluster ID: 42029c62df7e483889ebdab83dcc4c12
I20250114 03:12:20.258121  5809 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:12:20.258183  4138 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250114 03:12:20.258877  5809 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:12:20.258997  5809 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54: Loaded TSK: 0
I20250114 03:12:20.259104  5809 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:12:20.259490  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:20.261559  5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.262164  5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.262871  5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.263430  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:20.263523  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:20.263576  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:20.263589  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824340263590 us; error 0 us; skew 500 ppm
I20250114 03:12:20.264263  4138 webserver.cc:458] Webserver started at http://127.4.10.129:38475/ using document root <none> and password file <none>
I20250114 03:12:20.264346  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:20.264389  4138 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250114 03:12:20.264994  4138 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.265436  5826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.265576  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.265620  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root
uuid: "9c2b40255a11408fb99e70287305dfab"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.265682  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:20.271035  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:20.271303  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:20.271456  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:20.272265  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 03:12:20.272315  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
I20250114 03:12:20.272730  4138 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 03:12:20.272760  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.272780  5833 tablet_bootstrap.cc:492] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Bootstrap starting.
I20250114 03:12:20.273890  5833 tablet_bootstrap.cc:492] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 03:12:20.274117  5833 tablet_bootstrap.cc:492] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Bootstrap complete.
I20250114 03:12:20.274217  5833 ts_tablet_manager.cc:1397] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:20.274389  5833 raft_consensus.cc:357] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.274453  5833 raft_consensus.cc:738] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2b40255a11408fb99e70287305dfab, State: Initialized, Role: FOLLOWER
I20250114 03:12:20.274505  5833 consensus_queue.cc:260] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.274573  5833 raft_consensus.cc:397] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:12:20.274607  5833 raft_consensus.cc:491] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:12:20.274626  5833 raft_consensus.cc:3054] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:12:20.275197  5833 raft_consensus.cc:513] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.275255  5833 leader_election.cc:304] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2b40255a11408fb99e70287305dfab; no voters: 
I20250114 03:12:20.275341  5833 leader_election.cc:290] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [CANDIDATE]: Term 2 election: Requested vote from peers 
I20250114 03:12:20.275415  5877 raft_consensus.cc:2798] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:12:20.275461  5833 ts_tablet_manager.cc:1428] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20250114 03:12:20.275535  5877 raft_consensus.cc:695] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 2 LEADER]: Becoming Leader. State: Replica: 9c2b40255a11408fb99e70287305dfab, State: Running, Role: LEADER
I20250114 03:12:20.275592  5877 consensus_queue.cc:237] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } }
I20250114 03:12:20.276515  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.129:45635
I20250114 03:12:20.276547  5896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.129:45635 every 8 connection(s)
I20250114 03:12:20.276858  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:20.279070  5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.279695  5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.280237  5904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.280336  5897 heartbeater.cc:346] Connected to a master server at 127.4.10.190:45057
I20250114 03:12:20.280404  5897 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:20.280514  5897 heartbeater.cc:510] Master 127.4.10.190:45057 requested a full tablet report, sending...
I20250114 03:12:20.280869  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:20.280879  5761 ts_manager.cc:194] Registered new tserver with Master: 9c2b40255a11408fb99e70287305dfab (127.4.10.129:45635)
I20250114 03:12:20.281010  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250114 03:12:20.280979  5761 catalog_manager.cc:5526] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "9c2b40255a11408fb99e70287305dfab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2b40255a11408fb99e70287305dfab" member_type: VOTER last_known_addr { host: "127.4.10.129" port: 41015 } health_report { overall_health: HEALTHY } } }
W20250114 03:12:20.281040  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:20.281061  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824340281061 us; error 0 us; skew 500 ppm
I20250114 03:12:20.281802  4138 webserver.cc:458] Webserver started at http://127.4.10.130:41595/ using document root <none> and password file <none>
I20250114 03:12:20.281935  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:20.281992  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:20.282042  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:20.282299  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-1-root/instance:
uuid: "5fc2a06940014c6cb4224f665d2ef115"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.282337  5761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36630
I20250114 03:12:20.282734  5897 heartbeater.cc:502] Master 127.4.10.190:45057 was elected leader, sending a full tablet report...
I20250114 03:12:20.283310  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20250114 03:12:20.283845  5911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.283977  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.284021  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-1-root
uuid: "5fc2a06940014c6cb4224f665d2ef115"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.284067  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:20.317162  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:20.317446  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:20.317621  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:20.317828  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:20.317888  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.317931  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:20.317960  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.321172  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.130:42715
I20250114 03:12:20.321234  5974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.130:42715 every 8 connection(s)
I20250114 03:12:20.321578  4138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:12:20.323647  5978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:12:20.324230  5981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.324522  5975 heartbeater.cc:346] Connected to a master server at 127.4.10.190:45057
I20250114 03:12:20.324573  5975 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:20.324666  5975 heartbeater.cc:510] Master 127.4.10.190:45057 requested a full tablet report, sending...
W20250114 03:12:20.324951  5979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:12:20.324918  5761 ts_manager.cc:194] Registered new tserver with Master: 5fc2a06940014c6cb4224f665d2ef115 (127.4.10.130:42715)
I20250114 03:12:20.325342  5761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36634
I20250114 03:12:20.325547  4138 server_base.cc:1034] running on GCE node
I20250114 03:12:20.325678  4138 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250114 03:12:20.325716  4138 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250114 03:12:20.325729  4138 hybrid_clock.cc:648] HybridClock initialized: now 1736824340325729 us; error 0 us; skew 500 ppm
I20250114 03:12:20.326396  4138 webserver.cc:458] Webserver started at http://127.4.10.131:35553/ using document root <none> and password file <none>
I20250114 03:12:20.326470  4138 fs_manager.cc:362] Metadata directory not provided
I20250114 03:12:20.326527  4138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:12:20.326606  4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:12:20.326907  4138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-2-root/instance:
uuid: "4320c5f9547548b8a3bdb5e1cb5297a8"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.327800  4138 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20250114 03:12:20.328325  5986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.328486  4138 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.328533  4138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-2-root
uuid: "4320c5f9547548b8a3bdb5e1cb5297a8"
format_stamp: "Formatted at 2025-01-14 03:12:20 on dist-test-slave-ppl6"
I20250114 03:12:20.328596  4138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEKd910/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1736824303196632-4138-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:12:20.353184  4138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250114 03:12:20.353452  4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:12:20.353642  4138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:12:20.353854  4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:12:20.353919  4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.353958  4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:12:20.353984  4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250114 03:12:20.357370  4138 rpc_server.cc:307] RPC server started. Bound to: 127.4.10.131:42673
I20250114 03:12:20.357450  6050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.10.131:42673 every 8 connection(s)
I20250114 03:12:20.360714  6051 heartbeater.cc:346] Connected to a master server at 127.4.10.190:45057
I20250114 03:12:20.360775  6051 heartbeater.cc:463] Registering TS with master...
I20250114 03:12:20.360875  6051 heartbeater.cc:510] Master 127.4.10.190:45057 requested a full tablet report, sending...
I20250114 03:12:20.361090  5761 ts_manager.cc:194] Registered new tserver with Master: 4320c5f9547548b8a3bdb5e1cb5297a8 (127.4.10.131:42673)
I20250114 03:12:20.361425  5761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36642
I20250114 03:12:20.362000  4138 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004446149s
I20250114 03:12:20.369252  4138 test_util.cc:274] Using random seed: -304591006
I20250114 03:12:20.374102  4138 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250114 03:12:20.374176  4138 tablet_server.cc:178] TabletServer@127.4.10.129:0 shutting down...
I20250114 03:12:20.376566  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:20.376708  4138 tablet_replica.cc:331] stopping tablet replica
I20250114 03:12:20.376753  4138 raft_consensus.cc:2238] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 2 LEADER]: Raft consensus shutting down.
I20250114 03:12:20.376789  4138 raft_consensus.cc:2267] T 669fbe4edad14c21a08237148db6c835 P 9c2b40255a11408fb99e70287305dfab [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:20.388443  4138 tablet_server.cc:195] TabletServer@127.4.10.129:0 shutdown complete.
I20250114 03:12:20.389395  4138 tablet_server.cc:178] TabletServer@127.4.10.130:0 shutting down...
I20250114 03:12:20.392120  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:20.403438  4138 tablet_server.cc:195] TabletServer@127.4.10.130:0 shutdown complete.
I20250114 03:12:20.404243  4138 tablet_server.cc:178] TabletServer@127.4.10.131:0 shutting down...
I20250114 03:12:20.407030  4138 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250114 03:12:20.418555  4138 tablet_server.cc:195] TabletServer@127.4.10.131:0 shutdown complete.
I20250114 03:12:20.419425  4138 master.cc:537] Master@127.4.10.190:45057 shutting down...
I20250114 03:12:20.422112  4138 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 2 LEADER]: Raft consensus shutting down.
I20250114 03:12:20.422192  4138 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 0c8660b5854e430d9d2cad8e8cd6ea54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:12:20.422268  4138 tablet_replica.cc:331] stopping tablet replica
I20250114 03:12:20.434330  4138 master.cc:559] Master@127.4.10.190:45057 shutdown complete.
[       OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (314 ms)
[----------] 4 tests from TimestampAdvancementITest (37237 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (37238 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData

 1 FAILED TEST
I20250114 03:12:20.438968  4138 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5204: suppressed but not reported on 9 messages since previous log ~1 seconds ago
I20250114 03:12:20.439029  4138 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20250114 03:12:20.439059  4138 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:536: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20250114 03:12:20.439085  4138 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:643: suppressed but not reported on 7 messages since previous log ~1 seconds ago
I20250114 03:12:20.439113  4138 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 3 messages since previous log ~4 seconds ago
I20250114 03:12:20.439177  4138 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:345: suppressed but not reported on 574 messages since previous log ~37 seconds ago