[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250622 20:05:12.034101 22914 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.96.190:38711
I20250622 20:05:12.034377 22914 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:05:12.034492 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:12.039820 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.039829 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.039970 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:12.039975 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:12.040256 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:12.040307 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:12.040328 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622712040328 us; error 0 us; skew 500 ppm
I20250622 20:05:12.041276 22914 webserver.cc:469] Webserver started at http://127.22.96.190:41279/ using document root <none> and password file <none>
I20250622 20:05:12.041405 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:12.041442 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:12.041504 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:12.042428 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/master-0-root/instance:
uuid: "165c38b7034241a0ba14564d08263eb3"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.043632 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250622 20:05:12.044132 22927 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.044277 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.044345 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/master-0-root
uuid: "165c38b7034241a0ba14564d08263eb3"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.044404 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:12.060158 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:12.060387 22914 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:05:12.060484 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:12.060784 22933 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20250622 20:05:12.064420 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.190:38711
I20250622 20:05:12.064476 22979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.190:38711 every 8 connection(s)
I20250622 20:05:12.064949 22980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:12.067206 22980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3: Bootstrap starting.
I20250622 20:05:12.067976 22980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:12.068212 22980 log.cc:826] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3: Log is configured to *not* fsync() on all Append() calls
I20250622 20:05:12.068666 22980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3: No bootstrap required, opened a new log
I20250622 20:05:12.069747 22980 raft_consensus.cc:357] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165c38b7034241a0ba14564d08263eb3" member_type: VOTER }
I20250622 20:05:12.069835 22980 raft_consensus.cc:383] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:12.069866 22980 raft_consensus.cc:738] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165c38b7034241a0ba14564d08263eb3, State: Initialized, Role: FOLLOWER
I20250622 20:05:12.069979 22980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165c38b7034241a0ba14564d08263eb3" member_type: VOTER }
I20250622 20:05:12.070050 22980 raft_consensus.cc:397] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:05:12.070084 22980 raft_consensus.cc:491] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:05:12.070130 22980 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:12.070607 22980 raft_consensus.cc:513] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165c38b7034241a0ba14564d08263eb3" member_type: VOTER }
I20250622 20:05:12.070693 22980 leader_election.cc:304] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165c38b7034241a0ba14564d08263eb3; no voters:
I20250622 20:05:12.070851 22980 leader_election.cc:290] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:05:12.070919 22983 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:12.071069 22983 raft_consensus.cc:695] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 165c38b7034241a0ba14564d08263eb3, State: Running, Role: LEADER
I20250622 20:05:12.071115 22980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:05:12.071152 22983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165c38b7034241a0ba14564d08263eb3" member_type: VOTER }
I20250622 20:05:12.071436 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165c38b7034241a0ba14564d08263eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165c38b7034241a0ba14564d08263eb3" member_type: VOTER } }
I20250622 20:05:12.071513 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:12.071662 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165c38b7034241a0ba14564d08263eb3. Latest consensus state: current_term: 1 leader_uuid: "165c38b7034241a0ba14564d08263eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165c38b7034241a0ba14564d08263eb3" member_type: VOTER } }
I20250622 20:05:12.071784 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:12.071807 22987 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:05:12.072299 22987 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:05:12.072563 22914 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 20:05:12.073649 22987 catalog_manager.cc:1349] Generated new cluster ID: 7a6559c026874a23895eda0c89a4cfe0
I20250622 20:05:12.073698 22987 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:05:12.081337 22987 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:05:12.081815 22987 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:05:12.084908 22987 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3: Generated new TSK 0
I20250622 20:05:12.085052 22987 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:05:12.088316 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:12.089680 23002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.089802 23003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.089886 23005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:12.090049 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:12.090144 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:12.090179 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:12.090202 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622712090202 us; error 0 us; skew 500 ppm
I20250622 20:05:12.091055 22914 webserver.cc:469] Webserver started at http://127.22.96.129:34071/ using document root <none> and password file <none>
I20250622 20:05:12.091150 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:12.091199 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:12.091274 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:12.091521 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-0-root/instance:
uuid: "3ba5e279945c42bf8ccfa9413268ba24"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.092387 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:12.092875 23010 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.092999 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.093041 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-0-root
uuid: "3ba5e279945c42bf8ccfa9413268ba24"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.093082 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:12.109072 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:12.109519 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:12.109722 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:12.109980 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:12.110013 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.110045 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:12.110075 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.113552 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.129:46635
I20250622 20:05:12.113608 23073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.129:46635 every 8 connection(s)
I20250622 20:05:12.113967 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:12.115653 23080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.115840 23078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.115944 23077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:12.116552 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:12.116657 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:12.116688 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:12.116704 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622712116705 us; error 0 us; skew 500 ppm
I20250622 20:05:12.118077 22914 webserver.cc:469] Webserver started at http://127.22.96.130:42057/ using document root <none> and password file <none>
I20250622 20:05:12.118212 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:12.118278 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:12.118371 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:12.118706 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-1-root/instance:
uuid: "a98c6fe5c70f4075ab88d0df16729372"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.119411 23074 heartbeater.cc:344] Connected to a master server at 127.22.96.190:38711
I20250622 20:05:12.119505 23074 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:12.119673 23074 heartbeater.cc:507] Master 127.22.96.190:38711 requested a full tablet report, sending...
I20250622 20:05:12.120091 22944 ts_manager.cc:194] Registered new tserver with Master: 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635)
I20250622 20:05:12.120714 22944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43484
I20250622 20:05:12.123119 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20250622 20:05:12.123644 23086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.123792 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.123836 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-1-root
uuid: "a98c6fe5c70f4075ab88d0df16729372"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.123889 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:12.136121 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:12.136384 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:12.136559 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:12.136754 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:12.136781 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.136817 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:12.136849 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.140354 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.130:33869
I20250622 20:05:12.140415 23149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.130:33869 every 8 connection(s)
I20250622 20:05:12.140753 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:12.141939 23153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.142027 23155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:12.142199 23152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:12.143018 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:12.143115 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:12.143144 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:12.143167 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622712143167 us; error 0 us; skew 500 ppm
I20250622 20:05:12.143957 22914 webserver.cc:469] Webserver started at http://127.22.96.131:34847/ using document root <none> and password file <none>
I20250622 20:05:12.144054 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:12.144098 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:12.144150 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:12.144459 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-2-root/instance:
uuid: "9228796aa80a4254abdbb77b79833403"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.145948 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:12.146078 23150 heartbeater.cc:344] Connected to a master server at 127.22.96.190:38711
I20250622 20:05:12.146137 23150 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:12.146238 23150 heartbeater.cc:507] Master 127.22.96.190:38711 requested a full tablet report, sending...
I20250622 20:05:12.146461 22944 ts_manager.cc:194] Registered new tserver with Master: a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:12.146553 23161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.146665 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.146751 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-2-root
uuid: "9228796aa80a4254abdbb77b79833403"
format_stamp: "Formatted at 2025-06-22 20:05:12 on dist-test-slave-422w"
I20250622 20:05:12.146821 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:12.146997 22944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43492
I20250622 20:05:12.166086 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:12.166343 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:12.166512 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:12.166712 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:12.166739 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.166787 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:12.166815 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.170361 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.131:34591
I20250622 20:05:12.170449 23224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.131:34591 every 8 connection(s)
I20250622 20:05:12.173980 23225 heartbeater.cc:344] Connected to a master server at 127.22.96.190:38711
I20250622 20:05:12.174039 23225 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:12.174140 23225 heartbeater.cc:507] Master 127.22.96.190:38711 requested a full tablet report, sending...
I20250622 20:05:12.174381 22944 ts_manager.cc:194] Registered new tserver with Master: 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:12.174867 22944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43496
I20250622 20:05:12.174994 22914 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004462869s
I20250622 20:05:12.183856 22914 test_util.cc:276] Using random seed: -1662698452
I20250622 20:05:12.189980 22944 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43528:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 20:05:12.190235 22944 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:05:12.196237 23038 tablet_service.cc:1468] Processing CreateTablet for tablet e154e8b9b25c4561a5c344f4dca73dd8 (DEFAULT_TABLE table=test-workload [id=d053343f1c324d55a614d42ed13254c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:12.196405 23038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e154e8b9b25c4561a5c344f4dca73dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:12.196627 23189 tablet_service.cc:1468] Processing CreateTablet for tablet e154e8b9b25c4561a5c344f4dca73dd8 (DEFAULT_TABLE table=test-workload [id=d053343f1c324d55a614d42ed13254c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:12.196713 23189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e154e8b9b25c4561a5c344f4dca73dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:12.197952 23247 tablet_bootstrap.cc:492] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Bootstrap starting.
I20250622 20:05:12.198007 23114 tablet_service.cc:1468] Processing CreateTablet for tablet e154e8b9b25c4561a5c344f4dca73dd8 (DEFAULT_TABLE table=test-workload [id=d053343f1c324d55a614d42ed13254c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:12.198103 23114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e154e8b9b25c4561a5c344f4dca73dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:12.198431 23247 tablet_bootstrap.cc:654] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:12.198958 23247 tablet_bootstrap.cc:492] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: No bootstrap required, opened a new log
I20250622 20:05:12.198992 23247 ts_tablet_manager.cc:1397] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:12.199115 23247 raft_consensus.cc:357] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.199157 23247 raft_consensus.cc:383] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:12.199170 23247 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Initialized, Role: FOLLOWER
I20250622 20:05:12.199209 23247 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.199496 23074 heartbeater.cc:499] Master 127.22.96.190:38711 was elected leader, sending a full tablet report...
I20250622 20:05:12.199666 23250 tablet_bootstrap.cc:492] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Bootstrap starting.
I20250622 20:05:12.199785 23247 ts_tablet_manager.cc:1428] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 20:05:12.200117 23250 tablet_bootstrap.cc:654] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:12.200636 23250 tablet_bootstrap.cc:492] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: No bootstrap required, opened a new log
I20250622 20:05:12.200685 23250 ts_tablet_manager.cc:1397] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:12.200840 23250 raft_consensus.cc:357] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.200896 23250 raft_consensus.cc:383] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:12.200919 23250 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Initialized, Role: FOLLOWER
I20250622 20:05:12.200980 23250 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.201134 23250 ts_tablet_manager.cc:1428] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:12.201180 23150 heartbeater.cc:499] Master 127.22.96.190:38711 was elected leader, sending a full tablet report...
I20250622 20:05:12.203019 23246 tablet_bootstrap.cc:492] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Bootstrap starting.
I20250622 20:05:12.203406 23246 tablet_bootstrap.cc:654] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:12.203886 23246 tablet_bootstrap.cc:492] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: No bootstrap required, opened a new log
I20250622 20:05:12.203931 23246 ts_tablet_manager.cc:1397] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:12.204066 23246 raft_consensus.cc:357] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.204109 23246 raft_consensus.cc:383] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:12.204133 23246 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Initialized, Role: FOLLOWER
I20250622 20:05:12.204186 23246 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.204401 23246 ts_tablet_manager.cc:1428] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250622 20:05:12.204437 23225 heartbeater.cc:499] Master 127.22.96.190:38711 was elected leader, sending a full tablet report...
I20250622 20:05:12.212139 23252 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:05:12.212222 23252 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.212450 23252 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:12.215531 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24" is_pre_election: true
I20250622 20:05:12.215541 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403" is_pre_election: true
I20250622 20:05:12.215631 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 0.
I20250622 20:05:12.215639 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 0.
I20250622 20:05:12.216106 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:12.216212 23252 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:05:12.216244 23252 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:12.216266 23252 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:12.217023 23252 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.217232 23252 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 1 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:12.217317 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:12.217334 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:12.217403 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:12.217432 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:12.217939 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 1.
I20250622 20:05:12.218096 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:12.218137 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 1.
I20250622 20:05:12.218170 23252 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:12.218294 23252 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 1 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:12.218355 23252 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.219036 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 0 to 1, leader changed from <none> to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 1 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:12.226063 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 1 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:05:12.226063 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 1 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:05:12.226302 23252 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:05:12.226410 23252 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:05:12.242098 22914 test_util.cc:276] Using random seed: -1662640213
I20250622 20:05:12.247607 22914 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250622 20:05:12.545629 23254 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:12.545730 23254 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:12.546447 23254 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.546661 23254 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 2 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:12.550029 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:12.550029 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:12.550300 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:12.550402 23254 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250622 20:05:12.553114 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 2 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 1. Current term is 2. Ops: []
I20250622 20:05:12.553344 23256 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20250622 20:05:12.553468 23256 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250622 20:05:12.553500 23256 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250622 20:05:12.567713 23256 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 20:05:12.567787 23256 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:12.567857 23256 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.567983 23256 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:12.655457 23249 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:12.655560 23249 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:12.656260 23249 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.656468 23249 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 2 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:12.660231 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:12.660231 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:12.660387 23199 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 2: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:12.660995 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 2.
I20250622 20:05:12.661159 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:12.661266 23249 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:05:12.661387 23249 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 2 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:12.661445 23249 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.661898 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 1 to 2, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 2 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:12.666574 23254 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:12.666641 23254 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:05:12.667284 23254 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.667398 23254 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 3 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:12.667586 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:12.667598 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:12.667671 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:05:12.668169 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 3.
I20250622 20:05:12.668327 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:12.668421 23254 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 20:05:12.668525 23254 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 3 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:12.668568 23254 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.755754 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 2 to 3, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 3 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:12.759291 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 3 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 2. Current term is 3. Ops: []
I20250622 20:05:12.759531 23249 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250622 20:05:12.764412 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 3 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20250622 20:05:12.764645 23254 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250622 20:05:12.770327 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 3 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 2. Current term is 3. Ops: []
I20250622 20:05:12.770582 23297 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 41, Last known committed idx: 41, Time since last communication: 0.000s
I20250622 20:05:12.770679 23297 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 3.41 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 41
I20250622 20:05:12.778291 23048 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 20:05:12.778359 23048 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:12.778410 23048 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.41, Last appended by leader: 41, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.778499 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:05:12.779183 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 3 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 3 index: 41. (term mismatch)
I20250622 20:05:12.779243 23048 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 40
I20250622 20:05:12.779269 23048 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 2.41
I20250622 20:05:12.779295 23048 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:12.779510 23254 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250622 20:05:12.866680 23256 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:12.866782 23256 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:05:12.867465 23256 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.867620 23256 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 4 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:12.867846 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:12.867842 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:12.868125 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:12.868216 23256 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250622 20:05:12.885957 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 4 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 3. Current term is 4. Ops: []
I20250622 20:05:12.886238 23300 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250622 20:05:12.956133 23254 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 20:05:12.956233 23254 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:12.956305 23254 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.956419 23254 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:05:12.991566 23256 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:12.991667 23256 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 4 FOLLOWER]: Advancing to term 5
I20250622 20:05:12.992308 23256 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.992486 23256 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 5 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:12.992731 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 5 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:12.992811 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 3 FOLLOWER]: Advancing to term 5
I20250622 20:05:12.992813 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 5 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:12.992969 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 4 FOLLOWER]: Advancing to term 5
I20250622 20:05:12.993522 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 5.
I20250622 20:05:12.993561 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 5.
I20250622 20:05:12.993741 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:12.993856 23256 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 5 FOLLOWER]: Leader election won for term 5
I20250622 20:05:12.993899 23256 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 5 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:12.993952 23256 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:12.994388 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 3 to 5, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 5 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:13.091050 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 5 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
I20250622 20:05:13.091418 23256 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250622 20:05:13.107030 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 5 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
I20250622 20:05:13.107424 23256 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250622 20:05:13.208066 23254 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:13.208182 23254 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 5 FOLLOWER]: Advancing to term 6
I20250622 20:05:13.208864 23254 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.208998 23254 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 6 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:13.209292 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:13.209365 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:13.209585 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:13.209712 23254 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250622 20:05:13.225076 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 6 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 5. Current term is 6. Ops: []
I20250622 20:05:13.225450 23301 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 5.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250622 20:05:13.236004 23278 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 5 LEADER]: Stepping down as leader of term 5
I20250622 20:05:13.236120 23278 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:13.236202 23278 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.236330 23278 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 5 FOLLOWER]: Advancing to term 6
I20250622 20:05:13.288255 23249 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:13.288380 23249 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 5 FOLLOWER]: Advancing to term 6
I20250622 20:05:13.289074 23249 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.289244 23249 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 6 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:13.289476 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:13.289484 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:13.289554 23199 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 6: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:13.290162 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 6.
I20250622 20:05:13.290330 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:13.290412 23249 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 6 FOLLOWER]: Leader election won for term 6
I20250622 20:05:13.290529 23249 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 6 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:13.290588 23249 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.291023 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 5 to 6, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 6 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:13.312692 23254 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:13.312809 23254 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 6 FOLLOWER]: Advancing to term 7
I20250622 20:05:13.313516 23254 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.313694 23254 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 7 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:13.314006 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:13.314010 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:13.314108 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 6 FOLLOWER]: Advancing to term 7
I20250622 20:05:13.314730 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 7.
I20250622 20:05:13.314945 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:13.315075 23254 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 7 FOLLOWER]: Leader election won for term 7
I20250622 20:05:13.315160 23254 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 7 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:13.315222 23254 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.343911 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 6 to 7, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 7 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:13.374737 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 7 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 6. Current term is 7. Ops: []
I20250622 20:05:13.375037 23249 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250622 20:05:13.381400 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 7 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 6. Current term is 7. Ops: []
I20250622 20:05:13.381675 23302 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250622 20:05:13.381778 23302 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 7.43 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20250622 20:05:13.398236 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 7 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 5 index: 42. Preceding OpId from leader: term: 7 index: 43. (index mismatch)
I20250622 20:05:13.398530 23254 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250622 20:05:13.431212 23047 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 6 LEADER]: Stepping down as leader of term 6
I20250622 20:05:13.431304 23047 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:13.431375 23047 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 6.43, Last appended by leader: 43, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.431475 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 6 FOLLOWER]: Advancing to term 7
I20250622 20:05:13.432097 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 7 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 7 index: 43. (term mismatch)
I20250622 20:05:13.432139 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 42
I20250622 20:05:13.432152 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 6.43
I20250622 20:05:13.432201 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250622 20:05:13.432272 23249 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20250622 20:05:13.432439 23254 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250622 20:05:13.500013 23278 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:13.500102 23278 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 7 FOLLOWER]: Advancing to term 8
I20250622 20:05:13.500743 23278 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.500901 23278 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 8 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:13.501120 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:13.501127 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:13.501361 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:13.501451 23278 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250622 20:05:13.522953 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 8 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 7. Current term is 8. Ops: []
I20250622 20:05:13.523295 23298 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 7.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250622 20:05:13.616262 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 8 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 7. Current term is 8. Ops: []
I20250622 20:05:13.616540 23254 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 7.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250622 20:05:13.630128 23304 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 7 LEADER]: Stepping down as leader of term 7
I20250622 20:05:13.630220 23304 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:13.630285 23304 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.630431 23304 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 7 FOLLOWER]: Advancing to term 8
I20250622 20:05:13.641960 23278 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:13.642045 23278 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 8 FOLLOWER]: Advancing to term 9
I20250622 20:05:13.642668 23278 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.642819 23278 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 9 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:13.643072 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 9 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:13.643085 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 9 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:13.643209 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 8 FOLLOWER]: Advancing to term 9
I20250622 20:05:13.643795 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 9.
I20250622 20:05:13.643990 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:13.644095 23278 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 9 FOLLOWER]: Leader election won for term 9
I20250622 20:05:13.644157 23278 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 9 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:13.644207 23278 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.644681 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 7 to 9, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 9 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:13.726625 23249 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:13.726722 23249 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 7 FOLLOWER]: Advancing to term 8
I20250622 20:05:13.727427 23249 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.727589 23249 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 8 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:13.727831 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:13.727842 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:13.727962 23199 raft_consensus.cc:2366] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 8. Current term is 9.
I20250622 20:05:13.728127 23012 leader_election.cc:400] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 8 election: Vote denied by peer 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591) with higher term. Message: Invalid argument: T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 8. Current term is 9.
I20250622 20:05:13.728174 23012 leader_election.cc:403] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250622 20:05:13.728236 23249 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 8 FOLLOWER]: Advancing to term 9
I20250622 20:05:13.728762 23249 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591) with higher term. Message: Invalid argument: T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 8. Current term is 9.
I20250622 20:05:13.730019 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 9 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 9 index: 44. (index mismatch)
I20250622 20:05:13.730243 23278 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250622 20:05:13.762565 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 9 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 9 index: 44. (index mismatch)
I20250622 20:05:13.762843 23278 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250622 20:05:13.940047 23304 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:13.940174 23304 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 9 FOLLOWER]: Advancing to term 10
I20250622 20:05:13.941176 23304 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:13.941426 23304 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 10 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:13.941645 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:13.941651 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:13.941815 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 10 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 9. Current term is 10. Ops: []
I20250622 20:05:13.941960 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:13.942086 23304 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250622 20:05:13.942196 23301 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 9.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250622 20:05:14.027438 23305 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 9 LEADER]: Stepping down as leader of term 9
I20250622 20:05:14.027549 23305 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:14.027616 23305 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.027746 23305 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 9 FOLLOWER]: Advancing to term 10
I20250622 20:05:14.035385 23249 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:14.035465 23249 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 9 FOLLOWER]: Advancing to term 10
I20250622 20:05:14.036087 23249 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.036245 23249 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 10 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:14.036480 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:14.036480 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:14.036597 23199 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 10: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:14.037103 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 10.
I20250622 20:05:14.037266 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:14.037356 23249 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 10 FOLLOWER]: Leader election won for term 10
I20250622 20:05:14.037456 23249 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 10 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:14.037513 23249 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.037972 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 9 to 10, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 10 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:14.047873 23304 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:14.047948 23304 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 10 FOLLOWER]: Advancing to term 11
I20250622 20:05:14.048542 23304 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.048684 23304 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 11 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:14.048907 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 11 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:14.048916 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 11 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:14.049016 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 10 FOLLOWER]: Advancing to term 11
I20250622 20:05:14.049546 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 11.
I20250622 20:05:14.049706 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:14.049800 23304 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 11 FOLLOWER]: Leader election won for term 11
I20250622 20:05:14.049851 23304 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 11 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:14.049887 23304 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.064831 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 10 to 11, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 11 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:14.122512 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 11 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 10. Current term is 11. Ops: []
I20250622 20:05:14.122848 23249 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250622 20:05:14.125200 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 11 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 10. Current term is 11. Ops: []
I20250622 20:05:14.125458 23306 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250622 20:05:14.125530 23306 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 11.45 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 44
I20250622 20:05:14.128633 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 11 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 9 index: 44. Preceding OpId from leader: term: 11 index: 45. (index mismatch)
I20250622 20:05:14.128895 23304 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250622 20:05:14.131632 23047 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 10 LEADER]: Stepping down as leader of term 10
I20250622 20:05:14.131690 23047 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:14.131760 23047 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 10.45, Last appended by leader: 45, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.131856 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 10 FOLLOWER]: Advancing to term 11
I20250622 20:05:14.134320 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 11 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 10 index: 45. Preceding OpId from leader: term: 11 index: 45. (term mismatch)
I20250622 20:05:14.134367 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 44
I20250622 20:05:14.134400 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 10.45
I20250622 20:05:14.134420 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:14.134711 23304 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250622 20:05:14.139780 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.141053 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.143237 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.147465 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.155707 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.171938 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.204219 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.268514 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.396807 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.509851 23305 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:14.509958 23305 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 11 FOLLOWER]: Advancing to term 12
I20250622 20:05:14.510636 23305 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.510790 23305 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 12 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:14.511047 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:14.511057 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:14.511302 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:14.511416 23305 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250622 20:05:14.520466 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 12 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 11. Current term is 12. Ops: []
I20250622 20:05:14.520802 23304 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 11.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250622 20:05:14.520965 23298 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 11 LEADER]: Stepping down as leader of term 11
I20250622 20:05:14.521010 23298 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:14.521075 23298 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 11.45, Last appended by leader: 45, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.521157 23298 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 11 FOLLOWER]: Advancing to term 12
I20250622 20:05:14.521862 23249 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:14.521911 23249 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 11 FOLLOWER]: Advancing to term 12
I20250622 20:05:14.522426 23249 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.522550 23249 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 12 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:14.522722 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:14.522759 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:14.522864 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 12: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:14.523301 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 12.
I20250622 20:05:14.523497 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:14.523597 23249 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 12 FOLLOWER]: Leader election won for term 12
I20250622 20:05:14.523741 23249 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 12 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:14.523804 23249 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 11.45, Last appended by leader: 45, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.524243 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 11 to 12, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 12 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:14.620805 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 12 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 11 index: 45. Preceding OpId from leader: term: 12 index: 46. (index mismatch)
I20250622 20:05:14.621115 23249 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250622 20:05:14.622359 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 12 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 11 index: 45. Preceding OpId from leader: term: 12 index: 46. (index mismatch)
I20250622 20:05:14.622567 23308 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20250622 20:05:14.653088 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:14.723551 23309 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:14.723657 23309 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 12 FOLLOWER]: Advancing to term 13
I20250622 20:05:14.724365 23309 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.724558 23309 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 13 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:14.724748 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 13 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:14.724768 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 13 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:14.724954 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 12 FOLLOWER]: Advancing to term 13
I20250622 20:05:14.725569 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 13.
I20250622 20:05:14.725747 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:14.725839 23309 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 13 FOLLOWER]: Leader election won for term 13
I20250622 20:05:14.725893 23309 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 13 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:14.725948 23309 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.726415 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 12 to 13, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 13 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:14.726454 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 13 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 12. Current term is 13. Ops: []
I20250622 20:05:14.726838 23308 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 12.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250622 20:05:14.726953 23308 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 12 LEADER]: Stepping down as leader of term 12
I20250622 20:05:14.726979 23308 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:14.727033 23308 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.727109 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 12 FOLLOWER]: Advancing to term 13
I20250622 20:05:14.834172 23298 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:14.834332 23298 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 13 FOLLOWER]: Advancing to term 14
I20250622 20:05:14.835045 23298 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.835232 23298 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 14 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:14.835487 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 14 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:14.835491 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 14 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:14.835575 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 13 FOLLOWER]: Advancing to term 14
I20250622 20:05:14.836170 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 14.
I20250622 20:05:14.836402 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:14.836511 23298 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 14 FOLLOWER]: Leader election won for term 14
I20250622 20:05:14.836566 23298 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 14 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:14.836637 23298 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.837071 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 13 to 14, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 14 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:14.839699 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 14 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 13. Current term is 14. Ops: []
I20250622 20:05:14.839927 23309 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250622 20:05:14.840008 23309 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 14.47 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 46
I20250622 20:05:14.840049 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 14 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 13. Current term is 14. Ops: []
I20250622 20:05:14.840050 23309 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 13 LEADER]: Stepping down as leader of term 13
I20250622 20:05:14.840188 23309 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:14.840266 23309 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 13.47, Last appended by leader: 47, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:14.840387 23309 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 13 FOLLOWER]: Advancing to term 14
W20250622 20:05:14.840376 23305 consensus_queue.cc:1162] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3ba5e279945c42bf8ccfa9413268ba24" responder_term: 14 status { last_received { term: 12 index: 46 } last_committed_idx: 46 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 13. Current term is 14. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250622 20:05:14.921653 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 14 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 13 index: 47. Preceding OpId from leader: term: 14 index: 47. (term mismatch)
I20250622 20:05:14.921758 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 46
I20250622 20:05:14.921775 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 13.47
I20250622 20:05:14.921790 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:14.922012 23298 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250622 20:05:14.923084 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 14 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 12 index: 46. Preceding OpId from leader: term: 14 index: 47. (index mismatch)
I20250622 20:05:14.923306 23310 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20250622 20:05:15.024197 23308 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:15.024297 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 14 FOLLOWER]: Advancing to term 15
I20250622 20:05:15.025030 23308 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.025216 23308 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 15 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:15.025442 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 15 candidate_status { last_received { term: 14 index: 47 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:15.025442 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 15 candidate_status { last_received { term: 14 index: 47 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:15.025712 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:15.025813 23308 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20250622 20:05:15.043329 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 15 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 14. Current term is 15. Ops: []
I20250622 20:05:15.043605 23298 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 14.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250622 20:05:15.043728 23298 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 14 LEADER]: Stepping down as leader of term 14
I20250622 20:05:15.043763 23298 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:15.043831 23298 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 14.47, Last appended by leader: 47, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.043905 23298 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 14 FOLLOWER]: Advancing to term 15
I20250622 20:05:15.100965 23309 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:15.101069 23309 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 14 FOLLOWER]: Advancing to term 15
I20250622 20:05:15.101747 23309 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.101928 23309 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 15 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:15.102173 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 15 candidate_status { last_received { term: 14 index: 47 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:15.102253 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 15: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:15.102186 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 15 candidate_status { last_received { term: 14 index: 47 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:15.102865 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 15.
I20250622 20:05:15.103092 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:15.103193 23309 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 15 FOLLOWER]: Leader election won for term 15
I20250622 20:05:15.103247 23309 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 15 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:15.103300 23309 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 14.47, Last appended by leader: 47, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.103757 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 14 to 15, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 15 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:15.165372 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:15.169391 23308 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:15.169482 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 15 FOLLOWER]: Advancing to term 16
I20250622 20:05:15.170168 23308 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.170351 23308 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 16 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:15.170588 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 16 candidate_status { last_received { term: 14 index: 47 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:15.170585 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 16 candidate_status { last_received { term: 14 index: 47 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:15.170687 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 15 FOLLOWER]: Advancing to term 16
I20250622 20:05:15.171280 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 16.
I20250622 20:05:15.171483 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:15.171579 23308 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 16 FOLLOWER]: Leader election won for term 16
I20250622 20:05:15.171639 23308 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 16 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:15.171690 23308 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 14.47, Last appended by leader: 47, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.172087 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 15 to 16, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 16 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:15.181393 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 16 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 15. Current term is 16. Ops: []
I20250622 20:05:15.181648 23309 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250622 20:05:15.181748 23309 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 15 LEADER]: Stepping down as leader of term 15
I20250622 20:05:15.181784 23309 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:15.181838 23309 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 15.48, Last appended by leader: 48, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.181906 23309 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 15 FOLLOWER]: Advancing to term 16
I20250622 20:05:15.255599 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 16 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 14 index: 47. Preceding OpId from leader: term: 16 index: 48. (index mismatch)
I20250622 20:05:15.255877 23308 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250622 20:05:15.256847 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 16 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 15 index: 48. Preceding OpId from leader: term: 16 index: 48. (term mismatch)
I20250622 20:05:15.256907 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 47
I20250622 20:05:15.256932 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 15.48
I20250622 20:05:15.256958 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:15.257155 23311 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20250622 20:05:15.357095 23298 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:15.357195 23298 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 16 FOLLOWER]: Advancing to term 17
I20250622 20:05:15.357827 23298 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.357964 23298 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 17 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:15.358217 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 17 candidate_status { last_received { term: 16 index: 48 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:15.358217 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 17 candidate_status { last_received { term: 16 index: 48 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:15.358469 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:15.358561 23298 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20250622 20:05:15.366174 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 17 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 16. Current term is 17. Ops: []
I20250622 20:05:15.366472 23306 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 16.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250622 20:05:15.366611 23308 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 16 LEADER]: Stepping down as leader of term 16
I20250622 20:05:15.366662 23308 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:15.366739 23308 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 16.48, Last appended by leader: 48, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.366820 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 16 FOLLOWER]: Advancing to term 17
I20250622 20:05:15.443934 23309 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:15.444041 23309 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 16 FOLLOWER]: Advancing to term 17
I20250622 20:05:15.444741 23309 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.444900 23309 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 17 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:15.445161 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 17 candidate_status { last_received { term: 16 index: 48 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:15.445164 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 17 candidate_status { last_received { term: 16 index: 48 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:15.445310 23199 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 17: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:15.445951 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 17.
I20250622 20:05:15.446122 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:15.446229 23309 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 17 FOLLOWER]: Leader election won for term 17
I20250622 20:05:15.446288 23309 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 17 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:15.446352 23309 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 16.48, Last appended by leader: 48, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.446811 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 16 to 17, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 17 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:15.488687 23298 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:15.488793 23298 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 17 FOLLOWER]: Advancing to term 18
I20250622 20:05:15.489454 23298 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.489604 23298 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 18 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:15.489862 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 18 candidate_status { last_received { term: 16 index: 48 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:15.489874 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 18 candidate_status { last_received { term: 16 index: 48 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:15.489997 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 17 FOLLOWER]: Advancing to term 18
I20250622 20:05:15.490578 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 18.
I20250622 20:05:15.490774 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:15.490868 23298 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 18 FOLLOWER]: Leader election won for term 18
I20250622 20:05:15.490953 23298 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 18 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:15.491011 23298 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 16.48, Last appended by leader: 48, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.491590 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 17 to 18, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 18 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:15.526216 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 18 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 17. Current term is 18. Ops: []
I20250622 20:05:15.526511 23309 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250622 20:05:15.526623 23309 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 18.49 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 48
I20250622 20:05:15.526664 23305 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 17 LEADER]: Stepping down as leader of term 17
I20250622 20:05:15.526705 23305 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:15.526759 23305 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 17.49, Last appended by leader: 49, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.526834 23305 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 17 FOLLOWER]: Advancing to term 18
I20250622 20:05:15.601195 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 18 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 17 index: 49. Preceding OpId from leader: term: 18 index: 49. (term mismatch)
I20250622 20:05:15.601282 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 48
I20250622 20:05:15.601296 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 17.49
I20250622 20:05:15.601320 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:15.601665 23298 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250622 20:05:15.602795 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 18 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 16 index: 48. Preceding OpId from leader: term: 18 index: 49. (index mismatch)
I20250622 20:05:15.603039 23312 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250622 20:05:15.796036 23308 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:15.796135 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 18 FOLLOWER]: Advancing to term 19
I20250622 20:05:15.796777 23308 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.796921 23308 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 19 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:15.797148 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 19 candidate_status { last_received { term: 18 index: 49 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:15.797170 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 19 candidate_status { last_received { term: 18 index: 49 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:15.797390 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:15.797477 23308 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250622 20:05:15.801477 23305 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:15.801548 23305 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 18 FOLLOWER]: Advancing to term 19
I20250622 20:05:15.802146 23305 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.802277 23305 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 19 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:15.802490 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 19 candidate_status { last_received { term: 18 index: 49 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:15.802561 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 19: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:15.802496 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 19 candidate_status { last_received { term: 18 index: 49 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:15.802775 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:15.802860 23305 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250622 20:05:15.806746 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 19 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 18. Current term is 19. Ops: []
I20250622 20:05:15.806978 23310 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 18.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250622 20:05:15.807075 23310 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 18 LEADER]: Stepping down as leader of term 18
I20250622 20:05:15.807106 23310 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:15.807168 23310 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 18.49, Last appended by leader: 49, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.807248 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 18 FOLLOWER]: Advancing to term 19
I20250622 20:05:15.922142 23308 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:15.922266 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 19 FOLLOWER]: Advancing to term 20
I20250622 20:05:15.922996 23308 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.923174 23308 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 20 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:15.923395 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 20 candidate_status { last_received { term: 18 index: 49 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:15.923465 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 19 FOLLOWER]: Advancing to term 20
I20250622 20:05:15.923399 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 20 candidate_status { last_received { term: 18 index: 49 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:15.923533 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 19 FOLLOWER]: Advancing to term 20
I20250622 20:05:15.924149 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 20.
I20250622 20:05:15.924149 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 20.
I20250622 20:05:15.924360 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters:
I20250622 20:05:15.924445 23308 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 20 FOLLOWER]: Leader election won for term 20
I20250622 20:05:15.924572 23308 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 20 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:15.924649 23308 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 18.49, Last appended by leader: 49, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:15.925086 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 18 to 20, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 20 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:16.001624 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 20 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 18 index: 49. Preceding OpId from leader: term: 20 index: 50. (index mismatch)
I20250622 20:05:16.001938 23308 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250622 20:05:16.003113 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 20 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 18 index: 49. Preceding OpId from leader: term: 20 index: 50. (index mismatch)
I20250622 20:05:16.003304 23314 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250622 20:05:16.103837 23310 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:16.103945 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 20 FOLLOWER]: Advancing to term 21
I20250622 20:05:16.104627 23310 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.104766 23310 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 21 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:16.105084 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 21 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:16.105084 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 21 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:16.105219 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 20 FOLLOWER]: Advancing to term 21
I20250622 20:05:16.105794 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 21.
I20250622 20:05:16.105986 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:16.106101 23310 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 21 FOLLOWER]: Leader election won for term 21
I20250622 20:05:16.106164 23310 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 21 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:16.106223 23310 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 20.50, Last appended by leader: 50, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.106700 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 20 to 21, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 21 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:16.119258 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 21 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 20. Current term is 21. Ops: []
I20250622 20:05:16.119573 23313 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 20.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250622 20:05:16.119675 23313 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 20 LEADER]: Stepping down as leader of term 20
I20250622 20:05:16.119695 23313 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:16.119757 23313 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 20.50, Last appended by leader: 50, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.119834 23313 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 20 FOLLOWER]: Advancing to term 21
I20250622 20:05:16.165640 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:16.219184 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 21 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 20 index: 50. Preceding OpId from leader: term: 21 index: 51. (index mismatch)
I20250622 20:05:16.219511 23310 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250622 20:05:16.220520 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 21 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 20 index: 50. Preceding OpId from leader: term: 21 index: 51. (index mismatch)
I20250622 20:05:16.220822 23310 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250622 20:05:16.403599 23308 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:16.403693 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 21 FOLLOWER]: Advancing to term 22
I20250622 20:05:16.404304 23308 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.404456 23308 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 22 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:16.404655 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 22 candidate_status { last_received { term: 21 index: 51 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:16.404678 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 22 candidate_status { last_received { term: 21 index: 51 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:16.404925 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:16.405022 23308 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20250622 20:05:16.422479 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 22 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 21. Current term is 22. Ops: []
I20250622 20:05:16.422793 23298 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 21.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250622 20:05:16.422938 23310 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 21 LEADER]: Stepping down as leader of term 21
I20250622 20:05:16.423005 23310 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:16.423063 23310 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 21.51, Last appended by leader: 51, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.423152 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 21 FOLLOWER]: Advancing to term 22
I20250622 20:05:16.495520 23305 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:16.495635 23305 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 21 FOLLOWER]: Advancing to term 22
I20250622 20:05:16.496301 23305 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.496480 23305 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 22 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:16.496721 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 22 candidate_status { last_received { term: 21 index: 51 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:16.496721 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 22 candidate_status { last_received { term: 21 index: 51 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:16.496835 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 22: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:16.497326 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 22.
I20250622 20:05:16.497498 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:16.497633 23305 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 22 FOLLOWER]: Leader election won for term 22
I20250622 20:05:16.497786 23305 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 22 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:16.497848 23305 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 21.51, Last appended by leader: 51, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.498353 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 21 to 22, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 22 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:16.542197 23308 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:16.542312 23308 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 22 FOLLOWER]: Advancing to term 23
I20250622 20:05:16.543041 23308 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.543213 23308 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 23 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:16.543463 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 23 candidate_status { last_received { term: 21 index: 51 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:16.543465 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 23 candidate_status { last_received { term: 21 index: 51 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:16.543586 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 22 FOLLOWER]: Advancing to term 23
I20250622 20:05:16.544158 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 23.
I20250622 20:05:16.544340 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:16.544466 23308 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 23 FOLLOWER]: Leader election won for term 23
I20250622 20:05:16.544529 23308 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 23 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:16.544582 23308 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 21.51, Last appended by leader: 51, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.545044 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 22 to 23, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 23 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:16.574223 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 23 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 22. Current term is 23. Ops: []
I20250622 20:05:16.574558 23305 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250622 20:05:16.574668 23305 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 23.52 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I20250622 20:05:16.574708 23315 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 22 LEADER]: Stepping down as leader of term 22
I20250622 20:05:16.574766 23315 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:16.574842 23315 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 22.52, Last appended by leader: 52, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.574944 23315 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 22 FOLLOWER]: Advancing to term 23
I20250622 20:05:16.645530 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 23 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 22 index: 52. Preceding OpId from leader: term: 23 index: 52. (term mismatch)
I20250622 20:05:16.645613 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 51
I20250622 20:05:16.645628 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 22.52
I20250622 20:05:16.645642 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:16.645931 23308 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250622 20:05:16.647023 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 23 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 21 index: 51. Preceding OpId from leader: term: 23 index: 52. (index mismatch)
I20250622 20:05:16.647269 23316 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250622 20:05:16.747942 23298 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:16.748046 23298 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 23 FOLLOWER]: Advancing to term 24
I20250622 20:05:16.748749 23298 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.748894 23298 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 24 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:16.749179 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 24 candidate_status { last_received { term: 23 index: 52 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:16.749188 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 24 candidate_status { last_received { term: 23 index: 52 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:16.749302 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 23 FOLLOWER]: Advancing to term 24
I20250622 20:05:16.749871 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 24.
I20250622 20:05:16.750041 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:16.750144 23298 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 24 FOLLOWER]: Leader election won for term 24
I20250622 20:05:16.750196 23298 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 24 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:16.750252 23298 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 23.52, Last appended by leader: 52, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.750761 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 23 to 24, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 24 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:16.755923 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 24 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 23. Current term is 24. Ops: []
I20250622 20:05:16.756199 23316 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 23.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250622 20:05:16.756319 23316 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 23 LEADER]: Stepping down as leader of term 23
I20250622 20:05:16.756350 23316 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:16.756431 23316 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 23.52, Last appended by leader: 52, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:16.756515 23316 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 23 FOLLOWER]: Advancing to term 24
I20250622 20:05:16.854266 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 24 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 23 index: 52. Preceding OpId from leader: term: 24 index: 53. (index mismatch)
I20250622 20:05:16.854568 23298 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250622 20:05:16.855819 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 24 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 23 index: 52. Preceding OpId from leader: term: 24 index: 53. (index mismatch)
I20250622 20:05:16.856046 23298 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250622 20:05:17.079275 23316 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:17.079387 23316 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 24 FOLLOWER]: Advancing to term 25
I20250622 20:05:17.080066 23316 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.080235 23316 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 25 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:17.080524 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:17.080513 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:17.080785 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:17.080883 23316 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20250622 20:05:17.086087 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 25 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 24. Current term is 25. Ops: []
I20250622 20:05:17.086464 23317 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 24.53, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250622 20:05:17.086596 23310 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 24 LEADER]: Stepping down as leader of term 24
I20250622 20:05:17.086642 23310 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:17.086712 23310 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 24.53, Last appended by leader: 53, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.086789 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 24 FOLLOWER]: Advancing to term 25
I20250622 20:05:17.146082 23315 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:17.146195 23315 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 24 FOLLOWER]: Advancing to term 25
I20250622 20:05:17.146946 23315 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.147141 23315 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 25 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:17.147356 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:17.147372 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:17.147496 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 25: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:17.148247 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 25.
I20250622 20:05:17.148514 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:17.148633 23315 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 25 FOLLOWER]: Leader election won for term 25
I20250622 20:05:17.148749 23315 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 25 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:17.148798 23315 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 24.53, Last appended by leader: 53, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.149389 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 24 to 25, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 25 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:17.165916 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:17.221832 23316 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:17.221936 23316 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 25 FOLLOWER]: Advancing to term 26
I20250622 20:05:17.222609 23316 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.222759 23316 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 26 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:17.222985 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 26 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:17.223067 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 25 FOLLOWER]: Advancing to term 26
I20250622 20:05:17.223067 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 26 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:17.223785 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 26.
I20250622 20:05:17.223995 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:17.224100 23316 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 26 FOLLOWER]: Leader election won for term 26
I20250622 20:05:17.224171 23316 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 26 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:17.224238 23316 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 24.53, Last appended by leader: 53, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.224732 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 25 to 26, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 26 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:17.232597 23199 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 26 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 25. Current term is 26. Ops: []
I20250622 20:05:17.232898 23315 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250622 20:05:17.232999 23315 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 25 LEADER]: Stepping down as leader of term 25
I20250622 20:05:17.233018 23315 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:17.233088 23315 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 25.54, Last appended by leader: 54, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.233160 23315 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 25 FOLLOWER]: Advancing to term 26
I20250622 20:05:17.314175 23199 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 26 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 24 index: 53. Preceding OpId from leader: term: 26 index: 54. (index mismatch)
I20250622 20:05:17.314491 23316 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250622 20:05:17.315590 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 26 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 25 index: 54. Preceding OpId from leader: term: 26 index: 54. (term mismatch)
I20250622 20:05:17.315642 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 53
I20250622 20:05:17.315689 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 25.54
I20250622 20:05:17.315722 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:17.315888 23319 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20250622 20:05:17.416349 23310 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:17.416476 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 26 FOLLOWER]: Advancing to term 27
I20250622 20:05:17.417155 23310 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.417331 23310 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 27 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:17.417649 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 27 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:17.417662 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 27 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:17.417771 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 26 FOLLOWER]: Advancing to term 27
I20250622 20:05:17.418320 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 27.
I20250622 20:05:17.418540 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:17.418660 23310 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 27 FOLLOWER]: Leader election won for term 27
I20250622 20:05:17.418771 23310 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 27 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:17.418849 23310 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 26.54, Last appended by leader: 54, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.419364 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 26 to 27, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 27 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:17.432512 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 27 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 26. Current term is 27. Ops: []
I20250622 20:05:17.432801 23316 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 26.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250622 20:05:17.432904 23316 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 26 LEADER]: Stepping down as leader of term 26
I20250622 20:05:17.432935 23316 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:17.432999 23316 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 26.54, Last appended by leader: 54, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.433074 23316 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 26 FOLLOWER]: Advancing to term 27
I20250622 20:05:17.499866 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 27 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 26 index: 54. Preceding OpId from leader: term: 27 index: 55. (index mismatch)
I20250622 20:05:17.500188 23310 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250622 20:05:17.501204 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 27 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 26 index: 54. Preceding OpId from leader: term: 27 index: 55. (index mismatch)
I20250622 20:05:17.501379 23310 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250622 20:05:17.701355 23315 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:17.701468 23315 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 27 FOLLOWER]: Advancing to term 28
I20250622 20:05:17.702152 23315 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.702317 23315 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 28 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:17.702536 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 28 candidate_status { last_received { term: 27 index: 55 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:17.702558 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 28 candidate_status { last_received { term: 27 index: 55 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:17.702767 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:17.702867 23315 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20250622 20:05:17.706450 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 28 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 27. Current term is 28. Ops: []
I20250622 20:05:17.706736 23310 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 27.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250622 20:05:17.706862 23310 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 27 LEADER]: Stepping down as leader of term 27
I20250622 20:05:17.706926 23310 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:17.706995 23310 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 27.55, Last appended by leader: 55, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.707068 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 27 FOLLOWER]: Advancing to term 28
I20250622 20:05:17.804378 23315 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:17.804490 23315 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 28 FOLLOWER]: Advancing to term 29
I20250622 20:05:17.804982 23319 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:17.805050 23319 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 27 FOLLOWER]: Advancing to term 28
I20250622 20:05:17.805222 23315 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.805498 23315 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 29 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:17.805595 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 29 candidate_status { last_received { term: 27 index: 55 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:17.805684 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 29 candidate_status { last_received { term: 27 index: 55 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:17.805750 23199 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 28 FOLLOWER]: Advancing to term 29
I20250622 20:05:17.805923 23319 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.806033 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 28 FOLLOWER]: Advancing to term 29
I20250622 20:05:17.806123 23319 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 28 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:17.806198 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 28 candidate_status { last_received { term: 27 index: 55 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:17.806254 23124 raft_consensus.cc:2366] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 28. Current term is 29.
I20250622 20:05:17.806414 23013 leader_election.cc:400] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 28 election: Vote denied by peer a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869) with higher term. Message: Invalid argument: T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 28. Current term is 29.
I20250622 20:05:17.806430 23199 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 29.
I20250622 20:05:17.806473 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 28 candidate_status { last_received { term: 27 index: 55 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:17.806546 23198 raft_consensus.cc:2366] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 28. Current term is 29.
I20250622 20:05:17.806624 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:17.806483 23013 leader_election.cc:403] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20250622 20:05:17.806783 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 29.
I20250622 20:05:17.806825 23012 leader_election.cc:400] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 28 election: Vote denied by peer 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591) with higher term. Message: Invalid argument: T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 28. Current term is 29.
I20250622 20:05:17.806854 23315 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 FOLLOWER]: Leader election won for term 29
I20250622 20:05:17.806977 23315 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:17.806993 23319 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869) with higher term. Message: Invalid argument: T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 28. Current term is 29.
I20250622 20:05:17.807072 23315 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 27.55, Last appended by leader: 55, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:17.807493 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 27 to 29, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 29 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:17.911140 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 29 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 27 index: 55. Preceding OpId from leader: term: 29 index: 56. (index mismatch)
I20250622 20:05:17.911453 23315 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250622 20:05:17.912595 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 29 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 27 index: 55. Preceding OpId from leader: term: 29 index: 56. (index mismatch)
I20250622 20:05:17.912870 23315 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20250622 20:05:18.012785 23319 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 29 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:18.012892 23319 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 29 FOLLOWER]: Advancing to term 30
I20250622 20:05:18.013614 23319 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.013751 23310 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 29 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:18.013834 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 29 FOLLOWER]: Advancing to term 30
I20250622 20:05:18.013908 23319 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 30 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:18.014001 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 30 candidate_status { last_received { term: 29 index: 56 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:18.014003 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 30 candidate_status { last_received { term: 29 index: 56 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:18.014536 23310 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.014685 23310 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 30 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:18.014694 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 30: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:18.014839 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 30 candidate_status { last_received { term: 29 index: 56 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:18.014921 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:18.014838 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 30 candidate_status { last_received { term: 29 index: 56 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:18.015002 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 30: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:18.015115 23319 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20250622 20:05:18.015151 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:18.015216 23310 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20250622 20:05:18.021442 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 30 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 29. Current term is 30. Ops: []
I20250622 20:05:18.021770 23320 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 29.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250622 20:05:18.021885 23320 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 LEADER]: Stepping down as leader of term 29
I20250622 20:05:18.021922 23320 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:18.021984 23320 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 29.56, Last appended by leader: 56, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.022066 23320 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 29 FOLLOWER]: Advancing to term 30
I20250622 20:05:18.127722 23319 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:18.127842 23319 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 30 FOLLOWER]: Advancing to term 31
I20250622 20:05:18.128540 23319 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.128727 23319 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 31 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:18.128990 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 31 candidate_status { last_received { term: 29 index: 56 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:18.129086 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 30 FOLLOWER]: Advancing to term 31
I20250622 20:05:18.129069 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 31 candidate_status { last_received { term: 29 index: 56 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:18.129223 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 30 FOLLOWER]: Advancing to term 31
I20250622 20:05:18.129761 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 31.
I20250622 20:05:18.129968 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 31.
I20250622 20:05:18.129999 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:18.130086 23319 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 31 FOLLOWER]: Leader election won for term 31
I20250622 20:05:18.130224 23319 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 31 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:18.130299 23319 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 29.56, Last appended by leader: 56, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.130792 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 29 to 31, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 31 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:18.166177 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:18.212314 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 31 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 29 index: 56. Preceding OpId from leader: term: 31 index: 57. (index mismatch)
I20250622 20:05:18.212707 23319 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250622 20:05:18.213781 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 31 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 29 index: 56. Preceding OpId from leader: term: 31 index: 57. (index mismatch)
I20250622 20:05:18.214023 23319 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20250622 20:05:18.394852 23320 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:18.394970 23320 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 31 FOLLOWER]: Advancing to term 32
I20250622 20:05:18.395617 23320 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.395787 23320 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 32 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:18.396001 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:18.395999 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:18.396251 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:18.396342 23320 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250622 20:05:18.400702 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 32 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 31. Current term is 32. Ops: []
I20250622 20:05:18.400957 23323 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 31.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250622 20:05:18.401069 23323 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 31 LEADER]: Stepping down as leader of term 31
I20250622 20:05:18.401103 23323 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:18.401166 23323 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.401247 23323 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 31 FOLLOWER]: Advancing to term 32
I20250622 20:05:18.485477 23310 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:18.485580 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 31 FOLLOWER]: Advancing to term 32
I20250622 20:05:18.486287 23310 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.486471 23310 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 32 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:18.486742 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:18.486759 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:18.486858 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 32: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:18.487422 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 32.
I20250622 20:05:18.487639 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:18.487733 23310 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 32 FOLLOWER]: Leader election won for term 32
I20250622 20:05:18.487843 23310 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 32 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:18.487900 23310 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.488345 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 31 to 32, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 32 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:18.532876 23320 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:18.532979 23320 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 32 FOLLOWER]: Advancing to term 33
I20250622 20:05:18.533690 23320 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.533859 23320 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 33 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:18.534065 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 33 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:18.534065 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 33 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:18.534176 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 32 FOLLOWER]: Advancing to term 33
I20250622 20:05:18.534756 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 33.
I20250622 20:05:18.534955 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:18.535053 23320 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 33 FOLLOWER]: Leader election won for term 33
I20250622 20:05:18.535181 23320 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 33 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:18.535239 23320 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.535773 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 32 to 33, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 33 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:18.565063 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 33 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 32. Current term is 33. Ops: []
I20250622 20:05:18.565366 23310 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250622 20:05:18.565475 23310 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 32 LEADER]: Stepping down as leader of term 32
I20250622 20:05:18.565495 23310 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:18.565559 23310 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 32.58, Last appended by leader: 58, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.565630 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 32 FOLLOWER]: Advancing to term 33
I20250622 20:05:18.627254 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 33 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 31 index: 57. Preceding OpId from leader: term: 33 index: 58. (index mismatch)
I20250622 20:05:18.627543 23320 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250622 20:05:18.628597 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 33 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 32 index: 58. Preceding OpId from leader: term: 33 index: 58. (term mismatch)
I20250622 20:05:18.628669 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 57
I20250622 20:05:18.628690 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 32.58
I20250622 20:05:18.628710 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:18.628880 23325 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20250622 20:05:18.824779 23323 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 33 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:18.824884 23323 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 33 FOLLOWER]: Advancing to term 34
I20250622 20:05:18.825601 23323 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.825773 23323 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 34 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:18.825996 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 34 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:18.826009 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 34 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:18.826247 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:18.826340 23323 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20250622 20:05:18.840176 23310 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 33 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:18.840251 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 33 FOLLOWER]: Advancing to term 34
I20250622 20:05:18.840832 23310 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.840962 23310 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 34 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:18.841184 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 34 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:18.841212 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 34 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:18.841289 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 34: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:18.841445 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:18.841552 23310 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20250622 20:05:18.848747 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 34 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 33. Current term is 34. Ops: []
I20250622 20:05:18.848986 23325 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 33.58, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250622 20:05:18.849088 23325 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 33 LEADER]: Stepping down as leader of term 33
I20250622 20:05:18.849121 23325 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:18.849174 23325 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 33.58, Last appended by leader: 58, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.849251 23325 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 33 FOLLOWER]: Advancing to term 34
I20250622 20:05:18.941819 23323 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:18.941929 23323 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 34 FOLLOWER]: Advancing to term 35
I20250622 20:05:18.942581 23323 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.942737 23323 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 35 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:18.942989 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 35 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:18.942989 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 35 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:18.943118 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 34 FOLLOWER]: Advancing to term 35
I20250622 20:05:18.943123 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 34 FOLLOWER]: Advancing to term 35
I20250622 20:05:18.943831 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 35.
I20250622 20:05:18.943831 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 35.
I20250622 20:05:18.944085 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters:
I20250622 20:05:18.944187 23323 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 35 FOLLOWER]: Leader election won for term 35
I20250622 20:05:18.944324 23323 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 35 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:18.944378 23323 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 33.58, Last appended by leader: 58, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:18.944813 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 33 to 35, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 35 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:19.021296 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 35 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 33 index: 58. Preceding OpId from leader: term: 35 index: 59. (index mismatch)
I20250622 20:05:19.021631 23323 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250622 20:05:19.022944 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 35 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 33 index: 58. Preceding OpId from leader: term: 35 index: 59. (index mismatch)
I20250622 20:05:19.023135 23327 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20250622 20:05:19.123502 23310 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:19.123608 23310 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 35 FOLLOWER]: Advancing to term 36
I20250622 20:05:19.123951 23325 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:19.124022 23325 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 35 FOLLOWER]: Advancing to term 36
I20250622 20:05:19.124318 23310 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.124495 23310 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 36 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:19.124701 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:19.124701 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:19.124764 23325 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.124919 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 36: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:19.124933 23325 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 36 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:19.125056 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:19.125103 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:19.125120 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:19.125129 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 36: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:19.125259 23310 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20250622 20:05:19.125357 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:19.125447 23325 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20250622 20:05:19.134506 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 36 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 35. Current term is 36. Ops: []
I20250622 20:05:19.134738 23326 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 35.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250622 20:05:19.134856 23326 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 35 LEADER]: Stepping down as leader of term 35
I20250622 20:05:19.134922 23326 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:19.134979 23326 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 35.59, Last appended by leader: 59, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.135052 23326 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 35 FOLLOWER]: Advancing to term 36
I20250622 20:05:19.166464 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:19.238016 23325 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:19.238123 23325 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 36 FOLLOWER]: Advancing to term 37
I20250622 20:05:19.238833 23325 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.239017 23325 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 37 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:19.239249 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 37 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:19.239323 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 36 FOLLOWER]: Advancing to term 37
I20250622 20:05:19.239259 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 37 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:19.239393 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 36 FOLLOWER]: Advancing to term 37
I20250622 20:05:19.239914 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 37.
I20250622 20:05:19.239975 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 37.
I20250622 20:05:19.240124 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:19.240222 23325 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 37 FOLLOWER]: Leader election won for term 37
I20250622 20:05:19.240284 23325 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 37 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:19.240339 23325 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 35.59, Last appended by leader: 59, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.240828 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 35 to 37, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 37 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:19.336354 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 37 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 35 index: 59. Preceding OpId from leader: term: 37 index: 60. (index mismatch)
I20250622 20:05:19.336652 23325 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250622 20:05:19.337575 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 37 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 35 index: 59. Preceding OpId from leader: term: 37 index: 60. (index mismatch)
I20250622 20:05:19.337790 23325 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20250622 20:05:19.437912 23326 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 37 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:19.438019 23326 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 37 FOLLOWER]: Advancing to term 38
I20250622 20:05:19.438728 23326 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.438844 23328 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 37 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:19.438957 23328 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 37 FOLLOWER]: Advancing to term 38
I20250622 20:05:19.439046 23326 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 38 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:19.439121 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:19.439121 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:19.439694 23328 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.439831 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 38: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:19.439833 23328 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 38 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:19.439983 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:19.439981 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:19.440026 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:19.440099 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 38: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:19.440197 23326 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20250622 20:05:19.440310 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:19.440387 23328 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20250622 20:05:19.441118 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 38 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 37. Current term is 38. Ops: []
I20250622 20:05:19.441402 23325 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 37.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250622 20:05:19.441517 23325 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 37 LEADER]: Stepping down as leader of term 37
I20250622 20:05:19.441552 23325 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:19.441602 23325 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 37.60, Last appended by leader: 60, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.441668 23325 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 37 FOLLOWER]: Advancing to term 38
I20250622 20:05:19.563694 23326 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:19.563802 23326 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 38 FOLLOWER]: Advancing to term 39
I20250622 20:05:19.564446 23326 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.564625 23326 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 39 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:19.564848 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 39 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:19.564848 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 39 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:19.564954 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 38 FOLLOWER]: Advancing to term 39
I20250622 20:05:19.564982 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 38 FOLLOWER]: Advancing to term 39
I20250622 20:05:19.565675 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 39.
I20250622 20:05:19.565675 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 39.
I20250622 20:05:19.565956 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters:
I20250622 20:05:19.566083 23326 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 39 FOLLOWER]: Leader election won for term 39
I20250622 20:05:19.566133 23326 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 39 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:19.566196 23326 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 37.60, Last appended by leader: 60, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.566694 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 37 to 39, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 39 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:19.646155 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 39 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 37 index: 60. Preceding OpId from leader: term: 39 index: 61. (index mismatch)
I20250622 20:05:19.646422 23326 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250622 20:05:19.647552 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 39 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 37 index: 60. Preceding OpId from leader: term: 39 index: 61. (index mismatch)
I20250622 20:05:19.647801 23329 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250622 20:05:19.748633 23320 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:19.748734 23320 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 39 FOLLOWER]: Advancing to term 40
I20250622 20:05:19.749401 23320 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.749596 23320 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 40 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:19.749828 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:19.749823 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:19.750092 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:19.750203 23320 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20250622 20:05:19.765797 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 40 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 39. Current term is 40. Ops: []
I20250622 20:05:19.766083 23326 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 39.61, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250622 20:05:19.766208 23326 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 39 LEADER]: Stepping down as leader of term 39
I20250622 20:05:19.766244 23326 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:19.766304 23326 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 39.61, Last appended by leader: 61, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.766392 23326 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 39 FOLLOWER]: Advancing to term 40
I20250622 20:05:19.824201 23328 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:19.824316 23328 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 39 FOLLOWER]: Advancing to term 40
I20250622 20:05:19.825440 23328 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.825793 23328 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 40 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:19.825876 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:19.825974 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:19.826078 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 40: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:19.826592 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 40.
I20250622 20:05:19.826788 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:19.826881 23328 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 40 FOLLOWER]: Leader election won for term 40
I20250622 20:05:19.826961 23328 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 40 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:19.827031 23328 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 39.61, Last appended by leader: 61, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.827497 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 39 to 40, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 40 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:19.882293 23320 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:19.882372 23320 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 40 FOLLOWER]: Advancing to term 41
I20250622 20:05:19.883028 23320 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.883188 23320 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 41 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:19.883391 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 41 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:19.883404 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 41 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:19.883481 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 40 FOLLOWER]: Advancing to term 41
I20250622 20:05:19.884050 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 41.
I20250622 20:05:19.884218 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:19.884300 23320 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 41 FOLLOWER]: Leader election won for term 41
I20250622 20:05:19.884351 23320 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 41 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:19.884405 23320 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 39.61, Last appended by leader: 61, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.884857 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 40 to 41, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 41 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:19.925217 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 41 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 40. Current term is 41. Ops: []
I20250622 20:05:19.925516 23328 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250622 20:05:19.925621 23328 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 40 LEADER]: Stepping down as leader of term 40
I20250622 20:05:19.925657 23328 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:19.925711 23328 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 40.62, Last appended by leader: 62, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:19.925781 23328 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 40 FOLLOWER]: Advancing to term 41
I20250622 20:05:19.971371 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 41 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 40 index: 62. Preceding OpId from leader: term: 41 index: 62. (term mismatch)
I20250622 20:05:19.971472 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 61
I20250622 20:05:19.971488 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 40.62
I20250622 20:05:19.971504 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:19.971745 23320 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250622 20:05:19.973044 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 41 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 39 index: 61. Preceding OpId from leader: term: 41 index: 62. (index mismatch)
I20250622 20:05:19.973250 23320 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250622 20:05:20.073866 23327 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 41 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:20.073964 23327 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 41 FOLLOWER]: Advancing to term 42
I20250622 20:05:20.074646 23327 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.074801 23327 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 42 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:20.075034 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:20.075083 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:20.075284 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:20.075397 23327 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20250622 20:05:20.080411 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 42 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 41. Current term is 42. Ops: []
I20250622 20:05:20.080672 23330 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 41.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250622 20:05:20.080785 23330 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 41 LEADER]: Stepping down as leader of term 41
I20250622 20:05:20.080823 23330 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:20.080889 23330 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.080968 23330 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 41 FOLLOWER]: Advancing to term 42
I20250622 20:05:20.151913 23328 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 41 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:20.152015 23328 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 41 FOLLOWER]: Advancing to term 42
I20250622 20:05:20.152667 23328 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.152805 23328 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 42 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:20.153065 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:20.153065 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:20.153200 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 42: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:20.153780 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 42.
I20250622 20:05:20.153965 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:20.154057 23328 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 42 FOLLOWER]: Leader election won for term 42
I20250622 20:05:20.154127 23328 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 42 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:20.154196 23328 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.154660 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 41 to 42, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 42 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:20.171204 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:20.182629 23327 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:20.182721 23327 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 42 FOLLOWER]: Advancing to term 43
I20250622 20:05:20.183439 23327 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.183590 23327 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 43 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:20.183821 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 43 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:20.183826 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 43 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:20.183941 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 42 FOLLOWER]: Advancing to term 43
I20250622 20:05:20.184449 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 43.
I20250622 20:05:20.184643 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:20.184748 23327 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 43 FOLLOWER]: Leader election won for term 43
I20250622 20:05:20.184813 23327 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 43 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:20.184866 23327 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.185321 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 42 to 43, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 43 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:20.251751 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 43 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 42. Current term is 43. Ops: []
I20250622 20:05:20.252025 23328 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250622 20:05:20.252143 23328 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 42 LEADER]: Stepping down as leader of term 42
I20250622 20:05:20.252177 23328 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:20.252234 23328 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 42.63, Last appended by leader: 63, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.252300 23328 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 42 FOLLOWER]: Advancing to term 43
I20250622 20:05:20.272678 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 43 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 42 index: 63. Preceding OpId from leader: term: 43 index: 63. (term mismatch)
I20250622 20:05:20.272759 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 62
I20250622 20:05:20.272773 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 42.63
I20250622 20:05:20.272786 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:20.273033 23327 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250622 20:05:20.274012 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 43 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 41 index: 62. Preceding OpId from leader: term: 43 index: 63. (index mismatch)
I20250622 20:05:20.274266 23326 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20250622 20:05:20.752530 23328 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:20.752638 23328 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 43 FOLLOWER]: Advancing to term 44
I20250622 20:05:20.753284 23328 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.753437 23328 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 44 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:20.753460 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 44 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 43. Current term is 44. Ops: []
I20250622 20:05:20.753594 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 44 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:20.753625 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 44 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:20.753804 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:20.753891 23328 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20250622 20:05:20.754108 23331 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 43.63, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250622 20:05:20.754215 23331 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 43 LEADER]: Stepping down as leader of term 43
I20250622 20:05:20.754249 23331 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:20.754307 23331 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 43.63, Last appended by leader: 63, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.754367 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 43 FOLLOWER]: Advancing to term 44
I20250622 20:05:20.807467 23332 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:20.807571 23332 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 43 FOLLOWER]: Advancing to term 44
I20250622 20:05:20.808287 23332 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.808440 23332 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 44 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:20.808650 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 44 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:20.808657 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 44 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:20.808784 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 44: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:20.809267 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 44.
I20250622 20:05:20.809464 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:20.809561 23332 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 44 FOLLOWER]: Leader election won for term 44
I20250622 20:05:20.809655 23332 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 44 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:20.809697 23332 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 43.63, Last appended by leader: 63, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.810130 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 43 to 44, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 44 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:20.900347 23328 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:20.900465 23328 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 44 FOLLOWER]: Advancing to term 45
I20250622 20:05:20.901160 23328 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.901295 23328 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 45 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:20.901616 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 45 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:20.901628 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 45 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:20.901705 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 44 FOLLOWER]: Advancing to term 45
I20250622 20:05:20.902287 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 45.
I20250622 20:05:20.902514 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:20.902612 23328 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 45 FOLLOWER]: Leader election won for term 45
I20250622 20:05:20.902748 23328 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 45 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:20.902806 23328 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 43.63, Last appended by leader: 63, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.903311 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 44 to 45, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 45 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:20.919618 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 45 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 44. Current term is 45. Ops: []
I20250622 20:05:20.919896 23332 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250622 20:05:20.920002 23332 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 45.64 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 63
I20250622 20:05:20.920038 23333 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 44 LEADER]: Stepping down as leader of term 44
I20250622 20:05:20.920091 23333 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:20.920181 23333 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 44.64, Last appended by leader: 64, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:20.920267 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 44 FOLLOWER]: Advancing to term 45
I20250622 20:05:20.994572 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 45 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 44 index: 64. Preceding OpId from leader: term: 45 index: 64. (term mismatch)
I20250622 20:05:20.994657 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 63
I20250622 20:05:20.994673 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 44.64
I20250622 20:05:20.994688 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:20.994976 23328 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250622 20:05:20.996134 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 45 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 43 index: 63. Preceding OpId from leader: term: 45 index: 64. (index mismatch)
I20250622 20:05:20.996336 23338 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20250622 20:05:21.171454 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:21.296005 23331 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:21.296105 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 45 FOLLOWER]: Advancing to term 46
I20250622 20:05:21.296861 23331 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.297042 23331 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 46 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:21.297226 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 46 candidate_status { last_received { term: 45 index: 64 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:21.297231 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 46 candidate_status { last_received { term: 45 index: 64 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:21.297498 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:21.297609 23331 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20250622 20:05:21.306789 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 46 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 45. Current term is 46. Ops: []
I20250622 20:05:21.307029 23338 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 45.64, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250622 20:05:21.307143 23338 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 45 LEADER]: Stepping down as leader of term 45
I20250622 20:05:21.307184 23338 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:21.307260 23338 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 45.64, Last appended by leader: 64, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.307339 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 45 FOLLOWER]: Advancing to term 46
I20250622 20:05:21.371434 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:21.371546 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 45 FOLLOWER]: Advancing to term 46
I20250622 20:05:21.372288 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.372570 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 46 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:21.372682 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 46 candidate_status { last_received { term: 45 index: 64 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:21.372682 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 46 candidate_status { last_received { term: 45 index: 64 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:21.372835 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 46: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:21.373347 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 46.
I20250622 20:05:21.373522 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:21.373618 23333 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 46 FOLLOWER]: Leader election won for term 46
I20250622 20:05:21.373669 23333 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 46 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:21.373736 23333 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 45.64, Last appended by leader: 64, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.374186 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 45 to 46, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 46 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:21.418031 23331 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:21.418140 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 46 FOLLOWER]: Advancing to term 47
I20250622 20:05:21.418818 23331 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.419020 23331 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 47 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:21.419241 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 47 candidate_status { last_received { term: 45 index: 64 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:21.419241 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 47 candidate_status { last_received { term: 45 index: 64 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:21.419343 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 46 FOLLOWER]: Advancing to term 47
I20250622 20:05:21.419976 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 47.
I20250622 20:05:21.420179 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:21.420323 23331 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 47 FOLLOWER]: Leader election won for term 47
I20250622 20:05:21.420457 23331 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 47 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:21.420523 23331 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 45.64, Last appended by leader: 64, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.420991 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 46 to 47, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 47 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:21.465619 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 47 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 46. Current term is 47. Ops: []
I20250622 20:05:21.465911 23333 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250622 20:05:21.466015 23333 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 46 LEADER]: Stepping down as leader of term 46
I20250622 20:05:21.466045 23333 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:21.466110 23333 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 46.65, Last appended by leader: 65, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.466181 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 46 FOLLOWER]: Advancing to term 47
I20250622 20:05:21.511524 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 47 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 46 index: 65. Preceding OpId from leader: term: 47 index: 65. (term mismatch)
I20250622 20:05:21.511603 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 64
I20250622 20:05:21.511617 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 46.65
I20250622 20:05:21.511632 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:21.511852 23331 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250622 20:05:21.512874 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 47 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 45 index: 64. Preceding OpId from leader: term: 47 index: 65. (index mismatch)
I20250622 20:05:21.513070 23331 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20250622 20:05:21.737162 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:21.737274 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 47 FOLLOWER]: Advancing to term 48
I20250622 20:05:21.738058 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.738216 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 48 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:21.738420 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 48 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:21.738430 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 48 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:21.738656 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:21.738765 23333 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20250622 20:05:21.740221 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 48 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 47. Current term is 48. Ops: []
I20250622 20:05:21.740422 23331 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 47.65, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250622 20:05:21.740530 23331 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 47 LEADER]: Stepping down as leader of term 47
I20250622 20:05:21.740563 23331 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:21.740622 23331 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 47.65, Last appended by leader: 65, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.740700 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 47 FOLLOWER]: Advancing to term 48
I20250622 20:05:21.775913 23337 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:21.776034 23337 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 47 FOLLOWER]: Advancing to term 48
I20250622 20:05:21.776786 23337 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.776938 23337 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 48 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:21.777166 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 48 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:21.777194 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 48 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:21.777297 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 48: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:21.777741 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 48.
I20250622 20:05:21.777908 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:21.777998 23337 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 48 FOLLOWER]: Leader election won for term 48
I20250622 20:05:21.778048 23337 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 48 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:21.778105 23337 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 47.65, Last appended by leader: 65, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.778537 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 47 to 48, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 48 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:21.872671 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:21.872771 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 48 FOLLOWER]: Advancing to term 49
I20250622 20:05:21.873479 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.873641 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 49 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:21.873870 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 49 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:21.873884 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 49 candidate_status { last_received { term: 47 index: 65 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:21.873993 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 48 FOLLOWER]: Advancing to term 49
I20250622 20:05:21.874516 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 49.
I20250622 20:05:21.874703 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:21.874799 23333 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 49 FOLLOWER]: Leader election won for term 49
I20250622 20:05:21.874861 23333 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 49 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:21.874933 23333 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 47.65, Last appended by leader: 65, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.875373 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 48 to 49, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 49 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:21.884905 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 49 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 48. Current term is 49. Ops: []
I20250622 20:05:21.885145 23337 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250622 20:05:21.885241 23337 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 49.66 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 65
I20250622 20:05:21.885269 23338 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 48 LEADER]: Stepping down as leader of term 48
I20250622 20:05:21.885321 23338 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:21.885380 23338 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 48.66, Last appended by leader: 66, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:21.885460 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 48 FOLLOWER]: Advancing to term 49
I20250622 20:05:21.967798 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 49 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 48 index: 66. Preceding OpId from leader: term: 49 index: 66. (term mismatch)
I20250622 20:05:21.967876 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 65
I20250622 20:05:21.967897 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 48.66
I20250622 20:05:21.967917 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:21.968183 23333 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250622 20:05:21.969440 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 49 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 47 index: 65. Preceding OpId from leader: term: 49 index: 66. (index mismatch)
I20250622 20:05:21.969607 23333 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250622 20:05:22.147137 23338 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 49 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:22.147259 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 49 FOLLOWER]: Advancing to term 50
I20250622 20:05:22.147993 23338 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.148145 23338 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 50 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:22.148379 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 50 candidate_status { last_received { term: 49 index: 66 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:22.148391 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 50 candidate_status { last_received { term: 49 index: 66 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:22.148633 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:22.148725 23338 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20250622 20:05:22.168226 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 50 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 49. Current term is 50. Ops: []
I20250622 20:05:22.168567 23333 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 49.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250622 20:05:22.168691 23333 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 49 LEADER]: Stepping down as leader of term 49
I20250622 20:05:22.168722 23333 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:22.168797 23333 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 49.66, Last appended by leader: 66, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.168888 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 49 FOLLOWER]: Advancing to term 50
I20250622 20:05:22.171698 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:22.236040 23331 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 49 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:22.236147 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 49 FOLLOWER]: Advancing to term 50
I20250622 20:05:22.236853 23331 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.237023 23331 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 50 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:22.237218 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 50 candidate_status { last_received { term: 49 index: 66 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:22.237231 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 50 candidate_status { last_received { term: 49 index: 66 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:22.237330 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 50: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:22.237846 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 50.
I20250622 20:05:22.238006 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:22.238102 23331 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 50 FOLLOWER]: Leader election won for term 50
I20250622 20:05:22.238166 23331 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 50 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:22.238226 23331 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 49.66, Last appended by leader: 66, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.238698 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 49 to 50, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 50 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:22.253121 23338 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:22.253192 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 50 FOLLOWER]: Advancing to term 51
I20250622 20:05:22.253790 23338 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.253926 23338 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 51 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:22.254122 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 51 candidate_status { last_received { term: 49 index: 66 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:22.254146 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 51 candidate_status { last_received { term: 49 index: 66 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:22.254225 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 50 FOLLOWER]: Advancing to term 51
I20250622 20:05:22.254739 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 51.
I20250622 20:05:22.254922 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:22.255020 23338 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 51 FOLLOWER]: Leader election won for term 51
I20250622 20:05:22.255086 23338 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 51 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:22.255144 23338 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 49.66, Last appended by leader: 66, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.255595 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 50 to 51, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 51 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:22.317090 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 51 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 50. Current term is 51. Ops: []
I20250622 20:05:22.317364 23331 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250622 20:05:22.317452 23331 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 51.67 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 66
I20250622 20:05:22.317490 23339 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 50 LEADER]: Stepping down as leader of term 50
I20250622 20:05:22.317536 23339 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:22.317606 23339 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 50.67, Last appended by leader: 67, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.317687 23339 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 50 FOLLOWER]: Advancing to term 51
I20250622 20:05:22.371136 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 51 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 49 index: 66. Preceding OpId from leader: term: 51 index: 67. (index mismatch)
I20250622 20:05:22.371416 23338 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250622 20:05:22.372454 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 51 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 50 index: 67. Preceding OpId from leader: term: 51 index: 67. (term mismatch)
I20250622 20:05:22.372506 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 66
I20250622 20:05:22.372529 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 50.67
I20250622 20:05:22.372551 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:22.372748 23337 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20250622 20:05:22.472683 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 51 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:22.472781 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 51 FOLLOWER]: Advancing to term 52
I20250622 20:05:22.473491 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.473640 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 52 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:22.473825 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:22.473826 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:22.474062 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:22.474182 23333 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20250622 20:05:22.478344 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 52 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 51. Current term is 52. Ops: []
I20250622 20:05:22.478611 23341 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 51.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250622 20:05:22.478760 23338 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 51 LEADER]: Stepping down as leader of term 51
I20250622 20:05:22.478809 23338 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:22.478869 23338 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 51.67, Last appended by leader: 67, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.478973 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 51 FOLLOWER]: Advancing to term 52
I20250622 20:05:22.564778 23339 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 51 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:22.564891 23339 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 51 FOLLOWER]: Advancing to term 52
I20250622 20:05:22.565590 23339 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.565768 23339 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 52 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:22.566008 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:22.566011 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:22.566143 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 52: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:22.566746 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 52.
I20250622 20:05:22.566960 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:22.567054 23339 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 52 FOLLOWER]: Leader election won for term 52
I20250622 20:05:22.567124 23339 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 52 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:22.567183 23339 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 51.67, Last appended by leader: 67, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.567606 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 51 to 52, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 52 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:22.609652 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:22.609756 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 52 FOLLOWER]: Advancing to term 53
I20250622 20:05:22.610419 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.610571 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 53 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:22.610802 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 53 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:22.610836 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 53 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:22.610930 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 52 FOLLOWER]: Advancing to term 53
I20250622 20:05:22.611500 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 53.
I20250622 20:05:22.611724 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:22.611837 23333 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 53 FOLLOWER]: Leader election won for term 53
I20250622 20:05:22.611894 23333 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 53 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:22.611953 23333 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 51.67, Last appended by leader: 67, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.612388 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 52 to 53, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 53 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:22.647197 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 53 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 52. Current term is 53. Ops: []
I20250622 20:05:22.647534 23339 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250622 20:05:22.647655 23339 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 52 LEADER]: Stepping down as leader of term 52
I20250622 20:05:22.647687 23339 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:22.647748 23339 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 52.68, Last appended by leader: 68, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.647840 23339 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 52 FOLLOWER]: Advancing to term 53
I20250622 20:05:22.692934 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 53 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 51 index: 67. Preceding OpId from leader: term: 53 index: 68. (index mismatch)
I20250622 20:05:22.693204 23333 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250622 20:05:22.694248 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 53 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 52 index: 68. Preceding OpId from leader: term: 53 index: 68. (term mismatch)
I20250622 20:05:22.694299 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 67
I20250622 20:05:22.694322 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 52.68
I20250622 20:05:22.694350 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 53 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:22.694496 23333 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250622 20:05:22.890460 23339 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 53 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:22.890563 23339 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 53 FOLLOWER]: Advancing to term 54
I20250622 20:05:22.891270 23339 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.891424 23339 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 54 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:22.891656 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:22.891664 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:22.891916 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:22.892035 23339 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20250622 20:05:22.906395 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 54 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 53. Current term is 54. Ops: []
I20250622 20:05:22.906659 23342 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 53.68, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250622 20:05:22.906809 23333 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 53 LEADER]: Stepping down as leader of term 53
I20250622 20:05:22.906867 23333 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:22.906960 23333 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 53.68, Last appended by leader: 68, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.907044 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 53 FOLLOWER]: Advancing to term 54
I20250622 20:05:22.974110 23338 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 53 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:22.974212 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 53 FOLLOWER]: Advancing to term 54
I20250622 20:05:22.974879 23338 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.975071 23338 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 54 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:22.975291 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:22.975359 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 54: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:22.975317 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:22.975988 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 54.
I20250622 20:05:22.976172 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:22.976270 23338 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 54 FOLLOWER]: Leader election won for term 54
I20250622 20:05:22.976334 23338 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 54 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:22.976390 23338 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 53.68, Last appended by leader: 68, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:22.976824 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 53 to 54, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 54 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:23.029954 23339 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:23.030058 23339 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 54 FOLLOWER]: Advancing to term 55
I20250622 20:05:23.030699 23339 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.030872 23339 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 55 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:23.031109 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 55 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:23.031107 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 55 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:23.031212 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 54 FOLLOWER]: Advancing to term 55
I20250622 20:05:23.031745 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 55.
I20250622 20:05:23.031934 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:23.032037 23339 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 55 FOLLOWER]: Leader election won for term 55
I20250622 20:05:23.032102 23339 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 55 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:23.032147 23339 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 53.68, Last appended by leader: 68, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.032615 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 54 to 55, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 55 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:23.056617 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 55 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 54. Current term is 55. Ops: []
I20250622 20:05:23.056901 23338 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250622 20:05:23.057016 23338 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 54 LEADER]: Stepping down as leader of term 54
I20250622 20:05:23.057035 23338 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:23.057107 23338 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 54.69, Last appended by leader: 69, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.057179 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 54 FOLLOWER]: Advancing to term 55
I20250622 20:05:23.116276 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 55 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 53 index: 68. Preceding OpId from leader: term: 55 index: 69. (index mismatch)
I20250622 20:05:23.116631 23339 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250622 20:05:23.117564 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 55 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 54 index: 69. Preceding OpId from leader: term: 55 index: 69. (term mismatch)
I20250622 20:05:23.117631 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 68
I20250622 20:05:23.117653 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 54.69
I20250622 20:05:23.117684 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 55 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:23.117884 23339 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250622 20:05:23.171954 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:23.299084 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 55 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:23.299191 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 55 FOLLOWER]: Advancing to term 56
I20250622 20:05:23.300021 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.300144 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 56 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 55. Current term is 56. Ops: []
I20250622 20:05:23.300310 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 56 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:23.300380 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:23.300347 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:23.300367 23331 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 55.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250622 20:05:23.300603 23339 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 55 LEADER]: Stepping down as leader of term 55
I20250622 20:05:23.300621 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:23.300645 23339 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:23.300720 23333 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20250622 20:05:23.300750 23339 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 55.69, Last appended by leader: 69, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.300843 23339 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 55 FOLLOWER]: Advancing to term 56
I20250622 20:05:23.323730 23338 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 55 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:23.323836 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 55 FOLLOWER]: Advancing to term 56
I20250622 20:05:23.324491 23338 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.324668 23338 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 56 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:23.324892 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:23.324954 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:23.325093 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 56: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:23.325501 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 56.
I20250622 20:05:23.325726 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:23.325826 23338 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 56 FOLLOWER]: Leader election won for term 56
I20250622 20:05:23.325881 23338 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 56 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:23.325918 23338 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 55.69, Last appended by leader: 69, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.326448 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 55 to 56, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 56 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:23.404098 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 56 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 55 index: 69. Preceding OpId from leader: term: 56 index: 70. (index mismatch)
I20250622 20:05:23.404381 23338 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250622 20:05:23.405457 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 56 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 55 index: 69. Preceding OpId from leader: term: 56 index: 70. (index mismatch)
I20250622 20:05:23.405689 23344 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20250622 20:05:23.506438 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 56 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:23.506532 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 56 FOLLOWER]: Advancing to term 57
I20250622 20:05:23.507236 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.507388 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 57 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:23.507598 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:23.507603 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:23.507869 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:23.507982 23333 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20250622 20:05:23.518965 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 57 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 56. Current term is 57. Ops: []
I20250622 20:05:23.519224 23338 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 56.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250622 20:05:23.519469 23341 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 56 LEADER]: Stepping down as leader of term 56
I20250622 20:05:23.519505 23341 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:23.519568 23341 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 56.70, Last appended by leader: 70, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.519652 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 56 FOLLOWER]: Advancing to term 57
I20250622 20:05:23.599232 23339 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 56 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:23.599335 23339 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 56 FOLLOWER]: Advancing to term 57
I20250622 20:05:23.600044 23339 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.600216 23339 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 57 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:23.600450 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:23.600450 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 57 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:23.600566 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 57: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:23.601116 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 57.
I20250622 20:05:23.601356 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:23.601464 23339 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 57 FOLLOWER]: Leader election won for term 57
I20250622 20:05:23.601519 23339 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 57 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:23.601570 23339 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 56.70, Last appended by leader: 70, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.602072 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 56 to 57, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 57 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:23.625600 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:23.625705 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 57 FOLLOWER]: Advancing to term 58
I20250622 20:05:23.626340 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.626487 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 58 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:23.626708 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 58 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:23.626739 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 58 candidate_status { last_received { term: 56 index: 70 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:23.626852 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 57 FOLLOWER]: Advancing to term 58
I20250622 20:05:23.627424 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 58.
I20250622 20:05:23.627653 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:23.627771 23333 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 58 FOLLOWER]: Leader election won for term 58
I20250622 20:05:23.627889 23333 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 58 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:23.627948 23333 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 56.70, Last appended by leader: 70, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.628381 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 57 to 58, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 58 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:23.711755 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 58 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 57. Current term is 58. Ops: []
I20250622 20:05:23.712049 23339 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250622 20:05:23.712152 23339 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 58.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 70
I20250622 20:05:23.712188 23331 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 57 LEADER]: Stepping down as leader of term 57
I20250622 20:05:23.712229 23331 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 57 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:23.712289 23331 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 57.71, Last appended by leader: 71, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.712364 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 57 FOLLOWER]: Advancing to term 58
I20250622 20:05:23.713495 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 58 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 57 index: 71. Preceding OpId from leader: term: 58 index: 71. (term mismatch)
I20250622 20:05:23.713573 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 70
I20250622 20:05:23.713595 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 57.71
I20250622 20:05:23.713617 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:23.713862 23333 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250622 20:05:23.714973 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 58 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 56 index: 70. Preceding OpId from leader: term: 58 index: 71. (index mismatch)
I20250622 20:05:23.715171 23346 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250622 20:05:23.816274 23338 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 58 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:23.816370 23338 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 58 FOLLOWER]: Advancing to term 59
I20250622 20:05:23.817214 23338 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.817353 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 59 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 58. Current term is 59. Ops: []
I20250622 20:05:23.817382 23338 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 59 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:23.817538 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 59 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:23.817555 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 59 candidate_status { last_received { term: 58 index: 71 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:23.817636 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 58 FOLLOWER]: Advancing to term 59
I20250622 20:05:23.817679 23345 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 58.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250622 20:05:23.817768 23345 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 58 LEADER]: Stepping down as leader of term 58
I20250622 20:05:23.817792 23345 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 58 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:23.817838 23345 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 58.71, Last appended by leader: 71, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.817899 23345 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 58 FOLLOWER]: Advancing to term 59
I20250622 20:05:23.818220 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 59.
I20250622 20:05:23.818392 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:23.818488 23338 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 59 FOLLOWER]: Leader election won for term 59
I20250622 20:05:23.818543 23338 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 59 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:23.818586 23338 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 58.71, Last appended by leader: 71, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:23.819027 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 58 to 59, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 59 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:23.907441 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 59 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 58 index: 71. Preceding OpId from leader: term: 59 index: 72. (index mismatch)
I20250622 20:05:23.907742 23338 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250622 20:05:23.908717 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 59 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 58 index: 71. Preceding OpId from leader: term: 59 index: 72. (index mismatch)
I20250622 20:05:23.908910 23338 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20250622 20:05:24.094179 23331 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 59 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:24.094319 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 59 FOLLOWER]: Advancing to term 60
I20250622 20:05:24.095028 23331 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.095196 23331 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 60 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:24.095427 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 60 candidate_status { last_received { term: 59 index: 72 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:24.095427 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 60 candidate_status { last_received { term: 59 index: 72 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:24.095695 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:24.095795 23331 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20250622 20:05:24.106138 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 60 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 59. Current term is 60. Ops: []
I20250622 20:05:24.106389 23341 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 59.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250622 20:05:24.106505 23341 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 59 LEADER]: Stepping down as leader of term 59
I20250622 20:05:24.106525 23341 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:24.106587 23341 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 59.72, Last appended by leader: 72, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.106686 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 59 FOLLOWER]: Advancing to term 60
I20250622 20:05:24.113539 23345 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 59 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:24.113624 23345 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 59 FOLLOWER]: Advancing to term 60
I20250622 20:05:24.114224 23345 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.114372 23345 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 60 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:24.114578 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 60 candidate_status { last_received { term: 59 index: 72 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:24.114604 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 60 candidate_status { last_received { term: 59 index: 72 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:24.114686 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 60: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:24.115278 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 60.
I20250622 20:05:24.115456 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:24.115542 23345 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 60 FOLLOWER]: Leader election won for term 60
I20250622 20:05:24.115605 23345 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 60 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:24.115664 23345 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 59.72, Last appended by leader: 72, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.116106 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 59 to 60, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 60 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:24.172264 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:24.221565 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 60 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 59 index: 72. Preceding OpId from leader: term: 60 index: 73. (index mismatch)
I20250622 20:05:24.221900 23345 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250622 20:05:24.223044 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 60 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 59 index: 72. Preceding OpId from leader: term: 60 index: 73. (index mismatch)
I20250622 20:05:24.223215 23345 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20250622 20:05:24.324501 23331 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 60 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:24.324649 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 60 FOLLOWER]: Advancing to term 61
I20250622 20:05:24.325289 23331 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.325428 23331 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 61 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:24.325630 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 61 candidate_status { last_received { term: 60 index: 73 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:24.325630 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 61 candidate_status { last_received { term: 60 index: 73 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:24.325887 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:24.326000 23331 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20250622 20:05:24.327227 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 61 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 60. Current term is 61. Ops: []
I20250622 20:05:24.327414 23333 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 60.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250622 20:05:24.327522 23333 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 60 LEADER]: Stepping down as leader of term 60
I20250622 20:05:24.327558 23333 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 60 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:24.327617 23333 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 60.73, Last appended by leader: 73, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.327678 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 60 FOLLOWER]: Advancing to term 61
I20250622 20:05:24.415300 23341 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 60 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:24.415401 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 60 FOLLOWER]: Advancing to term 61
I20250622 20:05:24.416112 23341 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.416265 23341 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 61 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:24.416548 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 61 candidate_status { last_received { term: 60 index: 73 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:24.416618 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 61: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:24.416577 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 61 candidate_status { last_received { term: 60 index: 73 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:24.417197 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 61.
I20250622 20:05:24.417382 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:24.417477 23341 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 61 FOLLOWER]: Leader election won for term 61
I20250622 20:05:24.417542 23341 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 61 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:24.417583 23341 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 60.73, Last appended by leader: 73, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.418051 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 60 to 61, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 61 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:24.463749 23331 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 61 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:24.463860 23331 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 61 FOLLOWER]: Advancing to term 62
I20250622 20:05:24.464510 23331 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.464798 23331 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 62 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:24.464965 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 62 candidate_status { last_received { term: 60 index: 73 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:24.464967 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 62 candidate_status { last_received { term: 60 index: 73 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:24.465075 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 61 FOLLOWER]: Advancing to term 62
I20250622 20:05:24.465626 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 62.
I20250622 20:05:24.465811 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 62 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:24.465906 23331 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 62 FOLLOWER]: Leader election won for term 62
I20250622 20:05:24.466018 23331 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 62 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:24.466058 23331 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 60.73, Last appended by leader: 73, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.466564 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 61 to 62, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 62 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:24.523905 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 62 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 61. Current term is 62. Ops: []
I20250622 20:05:24.524247 23341 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250622 20:05:24.524400 23341 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 62.74 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 73
I20250622 20:05:24.524447 23344 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 61 LEADER]: Stepping down as leader of term 61
I20250622 20:05:24.524505 23344 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 61 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:24.524587 23344 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 61.74, Last appended by leader: 74, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.524688 23344 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 61 FOLLOWER]: Advancing to term 62
I20250622 20:05:24.551990 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 62 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 61 index: 74. Preceding OpId from leader: term: 62 index: 74. (term mismatch)
I20250622 20:05:24.552073 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 73
I20250622 20:05:24.552086 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 61.74
I20250622 20:05:24.552101 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 62 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:24.552372 23331 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250622 20:05:24.553546 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 62 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 60 index: 73. Preceding OpId from leader: term: 62 index: 74. (index mismatch)
I20250622 20:05:24.553762 23331 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250622 20:05:24.655144 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 62 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:24.655256 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 62 FOLLOWER]: Advancing to term 63
I20250622 20:05:24.655983 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.656143 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 63 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:24.656390 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 63 candidate_status { last_received { term: 62 index: 74 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:24.656391 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 63 candidate_status { last_received { term: 62 index: 74 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:24.656679 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 63 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:24.656782 23333 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 63 FOLLOWER]: Leader election lost for term 63. Reason: could not achieve majority
I20250622 20:05:24.663172 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 63 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 62. Current term is 63. Ops: []
I20250622 20:05:24.663425 23348 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 62.74, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250622 20:05:24.663537 23348 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 62 LEADER]: Stepping down as leader of term 62
I20250622 20:05:24.663561 23348 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 62 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:24.663666 23348 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 62.74, Last appended by leader: 74, Current term: 62, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.663743 23348 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 62 FOLLOWER]: Advancing to term 63
I20250622 20:05:24.749758 23344 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 62 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:24.749868 23344 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 62 FOLLOWER]: Advancing to term 63
I20250622 20:05:24.750574 23344 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.750747 23344 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 63 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:24.751010 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 63 candidate_status { last_received { term: 62 index: 74 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:24.751060 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 63 candidate_status { last_received { term: 62 index: 74 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:24.751165 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 63: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:24.751672 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 63.
I20250622 20:05:24.751855 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:24.751958 23344 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 63 FOLLOWER]: Leader election won for term 63
I20250622 20:05:24.752019 23344 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 63 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:24.752075 23344 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 62.74, Last appended by leader: 74, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.752499 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 62 to 63, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 63 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:24.806206 23333 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 63 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:24.806319 23333 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 63 FOLLOWER]: Advancing to term 64
I20250622 20:05:24.807049 23333 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.807228 23333 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 64 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:24.807461 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 64 candidate_status { last_received { term: 62 index: 74 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:24.807466 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 64 candidate_status { last_received { term: 62 index: 74 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:24.807533 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 63 FOLLOWER]: Advancing to term 64
I20250622 20:05:24.808369 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 64.
I20250622 20:05:24.808549 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:24.808666 23333 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 64 FOLLOWER]: Leader election won for term 64
I20250622 20:05:24.808739 23333 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 64 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:24.808794 23333 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 62.74, Last appended by leader: 74, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.809259 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 63 to 64, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 64 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:24.832329 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 64 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 63. Current term is 64. Ops: []
I20250622 20:05:24.832610 23344 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250622 20:05:24.832717 23344 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 64.75 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 74
I20250622 20:05:24.832741 23344 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 63 LEADER]: Stepping down as leader of term 63
I20250622 20:05:24.832774 23344 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 63 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:24.832844 23344 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 63.75, Last appended by leader: 75, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:24.832909 23344 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 63 FOLLOWER]: Advancing to term 64
I20250622 20:05:24.884593 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 64 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 62 index: 74. Preceding OpId from leader: term: 64 index: 75. (index mismatch)
I20250622 20:05:24.884876 23333 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250622 20:05:24.885607 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 64 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 63 index: 75. Preceding OpId from leader: term: 64 index: 75. (term mismatch)
I20250622 20:05:24.885711 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 74
I20250622 20:05:24.885761 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 63.75
I20250622 20:05:24.885797 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 64 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:24.885996 23333 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20250622 20:05:25.070587 23348 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 64 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:25.070689 23348 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 64 FOLLOWER]: Advancing to term 65
I20250622 20:05:25.071405 23348 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.071583 23348 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 65 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:25.071825 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 65 candidate_status { last_received { term: 64 index: 75 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:25.071825 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 65 candidate_status { last_received { term: 64 index: 75 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:25.072130 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 65 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:25.072232 23348 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 65 FOLLOWER]: Leader election lost for term 65. Reason: could not achieve majority
I20250622 20:05:25.075469 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 65 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 64. Current term is 65. Ops: []
I20250622 20:05:25.075654 23349 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 64.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250622 20:05:25.075753 23349 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 64 LEADER]: Stepping down as leader of term 64
I20250622 20:05:25.075790 23349 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 64 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:25.075845 23349 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 64.75, Last appended by leader: 75, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.075918 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 64 FOLLOWER]: Advancing to term 65
I20250622 20:05:25.087813 23344 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 64 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:25.087913 23344 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 64 FOLLOWER]: Advancing to term 65
I20250622 20:05:25.088567 23344 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.088724 23344 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 65 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:25.088979 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 65 candidate_status { last_received { term: 64 index: 75 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:25.089013 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 65 candidate_status { last_received { term: 64 index: 75 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:25.089054 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 65 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 65: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:25.089663 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 65.
I20250622 20:05:25.089847 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:25.089936 23344 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 65 FOLLOWER]: Leader election won for term 65
I20250622 20:05:25.089990 23344 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 65 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:25.090032 23344 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 64.75, Last appended by leader: 75, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.090461 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 64 to 65, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 65 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:25.169001 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 65 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 64 index: 75. Preceding OpId from leader: term: 65 index: 76. (index mismatch)
I20250622 20:05:25.169303 23344 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250622 20:05:25.170418 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 65 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 64 index: 75. Preceding OpId from leader: term: 65 index: 76. (index mismatch)
I20250622 20:05:25.170675 23350 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20250622 20:05:25.172502 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:25.271958 23348 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 65 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:25.272080 23348 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 65 FOLLOWER]: Advancing to term 66
I20250622 20:05:25.272727 23348 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.272864 23348 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 66 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:25.273053 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 66 candidate_status { last_received { term: 65 index: 76 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:25.273065 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 66 candidate_status { last_received { term: 65 index: 76 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:25.273290 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:25.273403 23348 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20250622 20:05:25.286374 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 66 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 65. Current term is 66. Ops: []
I20250622 20:05:25.286692 23341 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 65.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250622 20:05:25.286801 23341 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 65 LEADER]: Stepping down as leader of term 65
I20250622 20:05:25.286831 23341 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 65 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:25.286878 23341 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 65.76, Last appended by leader: 76, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.286979 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 65 FOLLOWER]: Advancing to term 66
I20250622 20:05:25.369756 23349 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 65 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:25.369875 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 65 FOLLOWER]: Advancing to term 66
I20250622 20:05:25.370615 23349 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.370806 23349 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 66 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:25.371054 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 66 candidate_status { last_received { term: 65 index: 76 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:25.371062 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 66 candidate_status { last_received { term: 65 index: 76 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:25.371161 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 66: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:25.371773 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 66 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 66.
I20250622 20:05:25.372007 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 66 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:25.372095 23349 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 66 FOLLOWER]: Leader election won for term 66
I20250622 20:05:25.372150 23349 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 66 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:25.372251 23349 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 65.76, Last appended by leader: 76, Current term: 66, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.372743 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 65 to 66, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 66 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:25.417989 23348 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:25.418093 23348 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 66 FOLLOWER]: Advancing to term 67
I20250622 20:05:25.418776 23348 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.418998 23348 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 67 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:25.419209 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 67 candidate_status { last_received { term: 65 index: 76 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:25.419225 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 67 candidate_status { last_received { term: 65 index: 76 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:25.419296 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 66 FOLLOWER]: Advancing to term 67
I20250622 20:05:25.419909 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 67.
I20250622 20:05:25.420153 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 67 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:25.420253 23348 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 67 FOLLOWER]: Leader election won for term 67
I20250622 20:05:25.420353 23348 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 67 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:25.420393 23348 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 65.76, Last appended by leader: 76, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.420933 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 66 to 67, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 67 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:25.455178 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 67 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 66. Current term is 67. Ops: []
I20250622 20:05:25.455479 23349 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250622 20:05:25.455624 23349 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 66 LEADER]: Stepping down as leader of term 66
I20250622 20:05:25.455668 23349 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 66 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:25.455735 23349 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 66.77, Last appended by leader: 77, Current term: 66, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.455806 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 66 FOLLOWER]: Advancing to term 67
I20250622 20:05:25.509450 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 67 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 66 index: 77. Preceding OpId from leader: term: 67 index: 77. (term mismatch)
I20250622 20:05:25.509532 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 76
I20250622 20:05:25.509562 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 66.77
I20250622 20:05:25.509593 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 67 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:25.509896 23348 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250622 20:05:25.510954 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 67 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 65 index: 76. Preceding OpId from leader: term: 67 index: 77. (index mismatch)
I20250622 20:05:25.511169 23351 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20250622 20:05:25.611728 23350 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 67 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:25.611840 23350 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 67 FOLLOWER]: Advancing to term 68
I20250622 20:05:25.612505 23350 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.612663 23350 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 68 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:25.612941 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 68 candidate_status { last_received { term: 67 index: 77 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:25.612941 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 68 candidate_status { last_received { term: 67 index: 77 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:25.613055 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 67 FOLLOWER]: Advancing to term 68
I20250622 20:05:25.613639 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 68.
I20250622 20:05:25.613830 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 68 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:25.613914 23350 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 68 FOLLOWER]: Leader election won for term 68
I20250622 20:05:25.613969 23350 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 68 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:25.614009 23350 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 67.77, Last appended by leader: 77, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.614544 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 67 to 68, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 68 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:25.625844 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 68 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 67. Current term is 68. Ops: []
I20250622 20:05:25.626118 23351 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 67.77, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250622 20:05:25.626233 23351 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 67 LEADER]: Stepping down as leader of term 67
I20250622 20:05:25.626252 23351 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 67 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:25.626325 23351 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 67.77, Last appended by leader: 77, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.626416 23351 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 67 FOLLOWER]: Advancing to term 68
I20250622 20:05:25.699081 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 68 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 67 index: 77. Preceding OpId from leader: term: 68 index: 78. (index mismatch)
I20250622 20:05:25.699378 23350 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250622 20:05:25.700470 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 68 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 67 index: 77. Preceding OpId from leader: term: 68 index: 78. (index mismatch)
I20250622 20:05:25.700691 23341 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20250622 20:05:25.801620 23349 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 68 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:25.801728 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 68 FOLLOWER]: Advancing to term 69
I20250622 20:05:25.802393 23349 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.802547 23349 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 69 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:25.802819 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 69 candidate_status { last_received { term: 68 index: 78 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:25.802827 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 69 candidate_status { last_received { term: 68 index: 78 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:25.802985 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 68 FOLLOWER]: Advancing to term 69
I20250622 20:05:25.803545 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 69.
I20250622 20:05:25.803745 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:25.803846 23349 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 69 FOLLOWER]: Leader election won for term 69
I20250622 20:05:25.803897 23349 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 69 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:25.803946 23349 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 68.78, Last appended by leader: 78, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.804399 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 69 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 68. Current term is 69. Ops: []
I20250622 20:05:25.804423 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 68 to 69, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 69 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:25.804618 23341 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 68.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250622 20:05:25.804710 23341 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 68 LEADER]: Stepping down as leader of term 68
I20250622 20:05:25.804744 23341 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 68 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:25.804793 23341 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 68.78, Last appended by leader: 78, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.804867 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 68 FOLLOWER]: Advancing to term 69
I20250622 20:05:25.904047 23348 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:25.904151 23348 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 69 FOLLOWER]: Advancing to term 70
I20250622 20:05:25.904829 23348 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.904992 23348 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 70 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:25.905236 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 70 candidate_status { last_received { term: 68 index: 78 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:25.905251 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 70 candidate_status { last_received { term: 68 index: 78 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:25.905372 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 69 FOLLOWER]: Advancing to term 70
I20250622 20:05:25.905934 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 70.
I20250622 20:05:25.906150 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:25.906248 23348 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 70 FOLLOWER]: Leader election won for term 70
I20250622 20:05:25.906301 23348 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 70 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:25.906340 23348 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 68.78, Last appended by leader: 78, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.906641 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 70 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 69. Current term is 70. Ops: []
I20250622 20:05:25.906842 23349 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250622 20:05:25.906822 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 69 to 70, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 70 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:25.906971 23349 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 69 LEADER]: Stepping down as leader of term 69
I20250622 20:05:25.907004 23349 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 69 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:25.907065 23349 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 69.79, Last appended by leader: 79, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:25.907130 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 69 FOLLOWER]: Advancing to term 70
I20250622 20:05:25.992252 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 70 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 68 index: 78. Preceding OpId from leader: term: 70 index: 79. (index mismatch)
I20250622 20:05:25.992532 23348 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250622 20:05:25.993572 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 70 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 69 index: 79. Preceding OpId from leader: term: 70 index: 79. (term mismatch)
I20250622 20:05:25.993642 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 78
I20250622 20:05:25.993685 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 69.79
I20250622 20:05:25.993710 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 70 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:25.993986 23351 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250622 20:05:26.172734 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:26.193603 23341 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 70 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:26.193701 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 70 FOLLOWER]: Advancing to term 71
I20250622 20:05:26.194386 23341 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.194518 23341 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 71 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:26.194728 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 71 candidate_status { last_received { term: 70 index: 79 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:26.194768 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 71 candidate_status { last_received { term: 70 index: 79 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:26.195004 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 71 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:26.195086 23341 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 71 FOLLOWER]: Leader election lost for term 71. Reason: could not achieve majority
I20250622 20:05:26.198604 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 71 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 70. Current term is 71. Ops: []
I20250622 20:05:26.198848 23352 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 70.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250622 20:05:26.198992 23352 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 70 LEADER]: Stepping down as leader of term 70
I20250622 20:05:26.199031 23352 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 70 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:26.199095 23352 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 70.79, Last appended by leader: 79, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.199174 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 70 FOLLOWER]: Advancing to term 71
I20250622 20:05:26.290405 23349 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 70 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:26.290509 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 70 FOLLOWER]: Advancing to term 71
I20250622 20:05:26.291198 23349 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.291349 23349 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 71 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:26.291574 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 71 candidate_status { last_received { term: 70 index: 79 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:26.291575 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 71 candidate_status { last_received { term: 70 index: 79 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:26.291695 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 71 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 71: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:26.292273 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 71.
I20250622 20:05:26.292479 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 71 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:26.292572 23349 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 71 FOLLOWER]: Leader election won for term 71
I20250622 20:05:26.292622 23349 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 71 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:26.292677 23349 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 70.79, Last appended by leader: 79, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.293134 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 70 to 71, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 71 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:26.312062 23341 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 71 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:26.312155 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 71 FOLLOWER]: Advancing to term 72
I20250622 20:05:26.312775 23341 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.312911 23341 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 72 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:26.313126 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 72 candidate_status { last_received { term: 70 index: 79 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:26.313189 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 71 FOLLOWER]: Advancing to term 72
I20250622 20:05:26.313169 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 72 candidate_status { last_received { term: 70 index: 79 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:26.313701 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 72.
I20250622 20:05:26.313858 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 72 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:26.313953 23341 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 72 FOLLOWER]: Leader election won for term 72
I20250622 20:05:26.314070 23341 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 72 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:26.314127 23341 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 70.79, Last appended by leader: 79, Current term: 72, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.314534 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 71 to 72, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 72 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:26.394111 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 72 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 70 index: 79. Preceding OpId from leader: term: 72 index: 80. (index mismatch)
I20250622 20:05:26.394407 23341 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250622 20:05:26.395711 23124 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 71 LEADER]: Stepping down as leader of term 71
I20250622 20:05:26.395776 23124 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 71 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:26.395828 23124 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 71.80, Last appended by leader: 80, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.395915 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 71 FOLLOWER]: Advancing to term 72
I20250622 20:05:26.396509 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 72 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 71 index: 80. Preceding OpId from leader: term: 72 index: 80. (term mismatch)
I20250622 20:05:26.396551 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 79
I20250622 20:05:26.396567 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 71.80
I20250622 20:05:26.396581 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 72 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:26.396875 23354 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250622 20:05:26.496268 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 72 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:26.496378 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 72 FOLLOWER]: Advancing to term 73
I20250622 20:05:26.497031 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.497193 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 73 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:26.497411 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 73 candidate_status { last_received { term: 72 index: 80 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:26.497524 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 72 FOLLOWER]: Advancing to term 73
I20250622 20:05:26.497408 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 73 candidate_status { last_received { term: 72 index: 80 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:26.498183 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 73.
I20250622 20:05:26.498278 23123 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 73 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 72. Current term is 73. Ops: []
I20250622 20:05:26.498365 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:26.498462 23352 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 73 FOLLOWER]: Leader election won for term 73
I20250622 20:05:26.498481 23354 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 72.80, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250622 20:05:26.498521 23352 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 73 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:26.498576 23354 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 72 LEADER]: Stepping down as leader of term 72
I20250622 20:05:26.498615 23354 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 72 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:26.498627 23352 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 72.80, Last appended by leader: 80, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.498662 23354 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 72.80, Last appended by leader: 80, Current term: 72, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.498728 23354 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 72 FOLLOWER]: Advancing to term 73
I20250622 20:05:26.498998 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 72 to 73, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 73 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:26.611510 23123 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 73 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 72 index: 80. Preceding OpId from leader: term: 73 index: 81. (index mismatch)
I20250622 20:05:26.611805 23352 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250622 20:05:26.612975 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 73 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 72 index: 80. Preceding OpId from leader: term: 73 index: 81. (index mismatch)
I20250622 20:05:26.613210 23352 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20250622 20:05:26.724442 23341 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 73 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:26.724545 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 73 FOLLOWER]: Advancing to term 74
I20250622 20:05:26.725260 23341 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.725404 23341 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 74 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:26.725643 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 74 candidate_status { last_received { term: 73 index: 81 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:26.725678 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 74 candidate_status { last_received { term: 73 index: 81 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:26.725898 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 74 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:26.725986 23341 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 74 FOLLOWER]: Leader election lost for term 74. Reason: could not achieve majority
I20250622 20:05:26.732847 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 74 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 73. Current term is 74. Ops: []
I20250622 20:05:26.733069 23351 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 73.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250622 20:05:26.733309 23352 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 73 LEADER]: Stepping down as leader of term 73
I20250622 20:05:26.733355 23352 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 73 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:26.733394 23352 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 73.81, Last appended by leader: 81, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.733476 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 73 FOLLOWER]: Advancing to term 74
I20250622 20:05:26.813884 23349 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 73 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:26.813988 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 73 FOLLOWER]: Advancing to term 74
I20250622 20:05:26.814931 23349 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.815119 23349 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 74 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:26.815323 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 74 candidate_status { last_received { term: 73 index: 81 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:26.815366 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 74 candidate_status { last_received { term: 73 index: 81 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:26.815460 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 74 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 74: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:26.816133 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 74.
I20250622 20:05:26.816376 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 74 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:26.816514 23349 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 74 FOLLOWER]: Leader election won for term 74
I20250622 20:05:26.816641 23349 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 74 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:26.816694 23349 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 73.81, Last appended by leader: 81, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.817142 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 73 to 74, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 74 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:26.839401 23341 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 74 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:26.839514 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 74 FOLLOWER]: Advancing to term 75
I20250622 20:05:26.840178 23341 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.840319 23341 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 75 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:26.840556 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 75 candidate_status { last_received { term: 73 index: 81 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:26.840569 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 75 candidate_status { last_received { term: 73 index: 81 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:26.840652 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 74 FOLLOWER]: Advancing to term 75
I20250622 20:05:26.841274 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 75 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 75.
I20250622 20:05:26.841454 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 75 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:26.841545 23341 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 75 FOLLOWER]: Leader election won for term 75
I20250622 20:05:26.841598 23341 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 75 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:26.841652 23341 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 73.81, Last appended by leader: 81, Current term: 75, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.842109 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 74 to 75, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 75 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:26.909507 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 75 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 74. Current term is 75. Ops: []
I20250622 20:05:26.909871 23349 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250622 20:05:26.909968 23349 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 75.82 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 81
I20250622 20:05:26.910017 23355 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 74 LEADER]: Stepping down as leader of term 74
I20250622 20:05:26.910090 23355 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 74 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:26.910172 23355 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 74.82, Last appended by leader: 82, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:26.910288 23355 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 74 FOLLOWER]: Advancing to term 75
I20250622 20:05:26.925367 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 75 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 73 index: 81. Preceding OpId from leader: term: 75 index: 82. (index mismatch)
I20250622 20:05:26.925704 23341 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250622 20:05:26.926789 23123 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 75 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 74 index: 82. Preceding OpId from leader: term: 75 index: 82. (term mismatch)
I20250622 20:05:26.926846 23123 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 81
I20250622 20:05:26.926864 23123 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 74.82
I20250622 20:05:26.926906 23123 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 75 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:26.927119 23354 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20250622 20:05:27.173000 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:27.191959 23355 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 75 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:27.192071 23355 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 75 FOLLOWER]: Advancing to term 76
I20250622 20:05:27.192766 23355 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.192950 23355 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 76 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:27.193161 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 76 candidate_status { last_received { term: 75 index: 82 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:27.193177 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 76 candidate_status { last_received { term: 75 index: 82 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:27.193393 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 76 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:27.193485 23355 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 76 FOLLOWER]: Leader election lost for term 76. Reason: could not achieve majority
I20250622 20:05:27.205936 23123 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 76 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 75. Current term is 76. Ops: []
I20250622 20:05:27.206241 23354 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 75.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250622 20:05:27.206357 23354 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 75 LEADER]: Stepping down as leader of term 75
I20250622 20:05:27.206394 23354 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 75 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:27.206450 23354 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 75.82, Last appended by leader: 82, Current term: 75, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.206537 23354 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 75 FOLLOWER]: Advancing to term 76
I20250622 20:05:27.285918 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 75 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:27.286026 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 75 FOLLOWER]: Advancing to term 76
I20250622 20:05:27.286696 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.286872 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 76 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:27.287127 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 76 candidate_status { last_received { term: 75 index: 82 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:27.287138 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 76 candidate_status { last_received { term: 75 index: 82 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:27.287225 23123 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 76: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:27.287765 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 76.
I20250622 20:05:27.287946 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 76 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:27.288028 23352 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 76 FOLLOWER]: Leader election won for term 76
I20250622 20:05:27.288131 23352 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 76 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:27.288175 23352 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 75.82, Last appended by leader: 82, Current term: 76, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.288619 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 75 to 76, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 76 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:27.320947 23355 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 76 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:27.321056 23355 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 76 FOLLOWER]: Advancing to term 77
I20250622 20:05:27.321777 23355 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.321996 23355 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 77 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:27.322207 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 77 candidate_status { last_received { term: 75 index: 82 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:27.322252 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 77 candidate_status { last_received { term: 75 index: 82 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:27.322368 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 76 FOLLOWER]: Advancing to term 77
I20250622 20:05:27.323088 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 77.
I20250622 20:05:27.323279 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 77 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:27.323362 23355 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 77 FOLLOWER]: Leader election won for term 77
I20250622 20:05:27.323418 23355 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 77 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:27.323468 23355 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 75.82, Last appended by leader: 82, Current term: 77, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.323987 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 76 to 77, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 77 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:27.365080 23123 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 77 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 76. Current term is 77. Ops: []
I20250622 20:05:27.365372 23352 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250622 20:05:27.365483 23352 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 77.83 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 82
I20250622 20:05:27.365525 23356 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 76 LEADER]: Stepping down as leader of term 76
I20250622 20:05:27.365572 23356 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 76 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:27.365648 23356 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 76.83, Last appended by leader: 83, Current term: 76, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.365741 23356 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 76 FOLLOWER]: Advancing to term 77
I20250622 20:05:27.406368 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 77 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 75 index: 82. Preceding OpId from leader: term: 77 index: 83. (index mismatch)
I20250622 20:05:27.406657 23355 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250622 20:05:27.407799 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 77 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 76 index: 83. Preceding OpId from leader: term: 77 index: 83. (term mismatch)
I20250622 20:05:27.407861 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 82
I20250622 20:05:27.407887 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 76.83
I20250622 20:05:27.407927 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 77 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:27.408143 23357 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250622 20:05:27.631510 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 77 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:27.631624 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 77 FOLLOWER]: Advancing to term 78
I20250622 20:05:27.632304 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.632472 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 78 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:27.632717 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 78 candidate_status { last_received { term: 77 index: 83 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:27.632717 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 78 candidate_status { last_received { term: 77 index: 83 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:27.632966 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 78 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:27.633056 23352 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 78 FOLLOWER]: Leader election lost for term 78. Reason: could not achieve majority
I20250622 20:05:27.647923 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 78 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 77. Current term is 78. Ops: []
I20250622 20:05:27.648216 23349 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 77.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250622 20:05:27.648346 23349 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 77 LEADER]: Stepping down as leader of term 77
I20250622 20:05:27.648380 23349 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 77 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:27.648444 23349 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 77.83, Last appended by leader: 83, Current term: 77, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.648522 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 77 FOLLOWER]: Advancing to term 78
I20250622 20:05:27.690233 23354 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 77 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:27.690339 23354 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 77 FOLLOWER]: Advancing to term 78
I20250622 20:05:27.691051 23354 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.691218 23354 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 78 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:27.691476 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 78 candidate_status { last_received { term: 77 index: 83 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:27.691476 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 78 candidate_status { last_received { term: 77 index: 83 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:27.691586 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 78 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 78: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:27.692338 23123 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 78.
I20250622 20:05:27.692534 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 78 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:27.692629 23354 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 78 FOLLOWER]: Leader election won for term 78
I20250622 20:05:27.692679 23354 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 78 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:27.692740 23354 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 77.83, Last appended by leader: 83, Current term: 78, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.693159 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 77 to 78, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 78 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:27.770809 23123 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 78 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 77 index: 83. Preceding OpId from leader: term: 78 index: 84. (index mismatch)
I20250622 20:05:27.771133 23354 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250622 20:05:27.772207 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 78 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 77 index: 83. Preceding OpId from leader: term: 78 index: 84. (index mismatch)
I20250622 20:05:27.772372 23354 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20250622 20:05:27.873889 23349 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 78 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:27.874023 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 78 FOLLOWER]: Advancing to term 79
I20250622 20:05:27.874702 23349 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.874877 23349 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 79 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:27.875123 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:27.875146 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:27.875366 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:27.875486 23349 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20250622 20:05:27.886437 23123 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 79 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 78. Current term is 79. Ops: []
I20250622 20:05:27.886745 23341 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 78.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250622 20:05:27.886862 23341 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 78 LEADER]: Stepping down as leader of term 78
I20250622 20:05:27.886926 23341 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 78 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:27.886992 23341 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 78.84, Last appended by leader: 84, Current term: 78, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.887070 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 78 FOLLOWER]: Advancing to term 79
I20250622 20:05:27.962638 23356 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 78 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:27.962760 23356 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 78 FOLLOWER]: Advancing to term 79
I20250622 20:05:27.963482 23356 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.963662 23356 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 79 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:27.963902 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:27.963902 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:27.964016 23123 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 79: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:27.964542 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 79.
I20250622 20:05:27.964762 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 79 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:27.964851 23356 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 79 FOLLOWER]: Leader election won for term 79
I20250622 20:05:27.964902 23356 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 79 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:27.964946 23356 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 78.84, Last appended by leader: 84, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:27.965369 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 78 to 79, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 79 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:28.016358 23349 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 79 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:28.016486 23349 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 79 FOLLOWER]: Advancing to term 80
I20250622 20:05:28.017423 23349 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.017580 23349 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 80 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:28.017829 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 80 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:28.017829 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 80 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:28.017972 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 79 FOLLOWER]: Advancing to term 80
I20250622 20:05:28.018564 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 80.
I20250622 20:05:28.018813 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:28.018950 23349 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 80 FOLLOWER]: Leader election won for term 80
I20250622 20:05:28.019003 23349 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 80 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:28.019062 23349 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 78.84, Last appended by leader: 84, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.019515 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 79 to 80, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 80 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:28.041865 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 80 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 79. Current term is 80. Ops: []
I20250622 20:05:28.042187 23356 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250622 20:05:28.042309 23356 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 79 LEADER]: Stepping down as leader of term 79
I20250622 20:05:28.042346 23356 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 79 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:28.042426 23356 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 79.85, Last appended by leader: 85, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.042508 23356 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 79 FOLLOWER]: Advancing to term 80
I20250622 20:05:28.113025 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 80 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 78 index: 84. Preceding OpId from leader: term: 80 index: 85. (index mismatch)
I20250622 20:05:28.113317 23349 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250622 20:05:28.114495 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 80 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 79 index: 85. Preceding OpId from leader: term: 80 index: 85. (term mismatch)
I20250622 20:05:28.114552 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 84
I20250622 20:05:28.114573 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 79.85
I20250622 20:05:28.114596 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 80 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:28.114784 23349 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20250622 20:05:28.173247 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:28.398541 23341 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 80 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:28.398665 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 80 FOLLOWER]: Advancing to term 81
I20250622 20:05:28.399413 23341 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.399565 23341 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 81 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:28.399888 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:28.399931 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:28.400137 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:28.400290 23341 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20250622 20:05:28.401829 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 81 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 80. Current term is 81. Ops: []
I20250622 20:05:28.402024 23359 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 80.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250622 20:05:28.402134 23359 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 80 LEADER]: Stepping down as leader of term 80
I20250622 20:05:28.402176 23359 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 80 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:28.402230 23359 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.402308 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 80 FOLLOWER]: Advancing to term 81
I20250622 20:05:28.471288 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 80 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:28.471396 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 80 FOLLOWER]: Advancing to term 81
I20250622 20:05:28.472149 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.472314 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 81 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:28.472522 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:28.472534 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:28.472645 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 81: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:28.473167 23123 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 81.
I20250622 20:05:28.473341 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 81 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:28.473435 23352 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 81 FOLLOWER]: Leader election won for term 81
I20250622 20:05:28.473498 23352 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 81 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:28.473546 23352 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.474031 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 80 to 81, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 81 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:28.545776 23341 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 81 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:28.545882 23341 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 81 FOLLOWER]: Advancing to term 82
I20250622 20:05:28.546521 23341 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.546669 23341 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 82 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:28.546972 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 82 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:28.546931 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 82 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:28.547048 23123 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 81 FOLLOWER]: Advancing to term 82
I20250622 20:05:28.547648 23123 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 82.
I20250622 20:05:28.547837 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 82 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:28.547935 23341 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 82 FOLLOWER]: Leader election won for term 82
I20250622 20:05:28.548065 23341 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 82 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:28.548125 23341 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.548554 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 81 to 82, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 82 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:28.549127 23123 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 82 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 81. Current term is 82. Ops: []
I20250622 20:05:28.549360 23352 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250622 20:05:28.549451 23352 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 81 LEADER]: Stepping down as leader of term 81
I20250622 20:05:28.549484 23352 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 81 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:28.549535 23352 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 81.86, Last appended by leader: 86, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.549578 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 81 FOLLOWER]: Advancing to term 82
I20250622 20:05:28.646152 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 82 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 81 index: 86. Preceding OpId from leader: term: 82 index: 86. (term mismatch)
I20250622 20:05:28.646235 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 85
I20250622 20:05:28.646250 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 81.86
I20250622 20:05:28.646263 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 82 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:28.646463 23341 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250622 20:05:28.647572 23123 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 82 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 80 index: 85. Preceding OpId from leader: term: 82 index: 86. (index mismatch)
I20250622 20:05:28.647748 23341 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250622 20:05:28.836390 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 82 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:28.836488 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 82 FOLLOWER]: Advancing to term 83
I20250622 20:05:28.837122 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.837272 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 83 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:28.837502 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:28.837507 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:28.837766 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 83 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:28.837878 23352 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 83 FOLLOWER]: Leader election lost for term 83. Reason: could not achieve majority
I20250622 20:05:28.838074 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 83 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 82. Current term is 83. Ops: []
I20250622 20:05:28.838248 23341 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 82.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250622 20:05:28.838353 23360 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 82 LEADER]: Stepping down as leader of term 82
I20250622 20:05:28.838403 23360 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 82 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:28.838472 23360 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.838562 23360 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 82 FOLLOWER]: Advancing to term 83
I20250622 20:05:28.907951 23357 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 82 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:28.908061 23357 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 82 FOLLOWER]: Advancing to term 83
I20250622 20:05:28.908840 23357 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.908998 23357 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 83 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:28.909251 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:28.909324 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 83 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 83: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:28.909263 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:28.909941 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 83.
I20250622 20:05:28.910158 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:28.910250 23357 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 83 FOLLOWER]: Leader election won for term 83
I20250622 20:05:28.910305 23357 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 83 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:28.910365 23357 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.910835 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 82 to 83, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 83 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:28.952515 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 83 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:28.952620 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 83 FOLLOWER]: Advancing to term 84
I20250622 20:05:28.953286 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.953438 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 84 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:28.953665 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 84 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:28.953667 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 84 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:28.953784 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 83 FOLLOWER]: Advancing to term 84
I20250622 20:05:28.954332 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 84.
I20250622 20:05:28.954510 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 84 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:28.954636 23352 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 84 FOLLOWER]: Leader election won for term 84
I20250622 20:05:28.954732 23352 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 84 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:28.954792 23352 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.955423 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 83 to 84, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 84 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:28.996495 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 84 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 83. Current term is 84. Ops: []
I20250622 20:05:28.996812 23357 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250622 20:05:28.996913 23357 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 84.87 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 86
I20250622 20:05:28.996966 23359 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 83 LEADER]: Stepping down as leader of term 83
I20250622 20:05:28.997006 23359 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 83 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:28.997062 23359 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 83.87, Last appended by leader: 87, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:28.997128 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 83 FOLLOWER]: Advancing to term 84
I20250622 20:05:29.041847 23123 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 84 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 83 index: 87. Preceding OpId from leader: term: 84 index: 87. (term mismatch)
I20250622 20:05:29.041931 23123 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 86
I20250622 20:05:29.041953 23123 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 83.87
I20250622 20:05:29.041968 23123 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 84 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:29.042184 23352 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250622 20:05:29.043390 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 84 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 82 index: 86. Preceding OpId from leader: term: 84 index: 87. (index mismatch)
I20250622 20:05:29.043584 23361 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20250622 20:05:29.173480 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:29.228039 23360 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 84 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:29.228129 23360 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 84 FOLLOWER]: Advancing to term 85
I20250622 20:05:29.229568 23360 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.229756 23360 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 85 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:29.229998 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:29.230018 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:29.230237 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 85 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:29.230327 23360 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 85 FOLLOWER]: Leader election lost for term 85. Reason: could not achieve majority
I20250622 20:05:29.232656 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 85 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 84. Current term is 85. Ops: []
I20250622 20:05:29.232919 23352 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 84.87, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250622 20:05:29.233031 23352 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 84 LEADER]: Stepping down as leader of term 84
I20250622 20:05:29.233064 23352 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 84 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:29.233111 23352 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.233181 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 84 FOLLOWER]: Advancing to term 85
I20250622 20:05:29.327080 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 84 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:29.327183 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 84 FOLLOWER]: Advancing to term 85
I20250622 20:05:29.327821 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.327972 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 85 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:29.328219 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:29.328296 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 85 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 85: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:29.328225 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:29.328903 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 85.
I20250622 20:05:29.329087 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:29.329177 23359 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 85 FOLLOWER]: Leader election won for term 85
I20250622 20:05:29.329238 23359 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 85 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:29.329293 23359 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.329718 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 84 to 85, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 85 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:29.378266 23360 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 85 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:29.378376 23360 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 85 FOLLOWER]: Advancing to term 86
I20250622 20:05:29.379144 23360 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.379290 23360 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 86 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:29.379539 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 86 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:29.379567 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 86 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:29.379637 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 85 FOLLOWER]: Advancing to term 86
I20250622 20:05:29.380239 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 86 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 86.
I20250622 20:05:29.380431 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 86 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:29.380524 23360 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 86 FOLLOWER]: Leader election won for term 86
I20250622 20:05:29.380632 23360 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 86 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:29.380676 23360 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 86, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.381124 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 85 to 86, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 86 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:29.440240 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 86 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 85. Current term is 86. Ops: []
I20250622 20:05:29.440513 23359 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250622 20:05:29.440654 23359 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 85 LEADER]: Stepping down as leader of term 85
I20250622 20:05:29.440696 23359 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 85 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:29.440768 23359 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 85.88, Last appended by leader: 88, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.440845 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 85 FOLLOWER]: Advancing to term 86
I20250622 20:05:29.464628 23123 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 86 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 85 index: 88. Preceding OpId from leader: term: 86 index: 88. (term mismatch)
I20250622 20:05:29.464726 23123 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 87
I20250622 20:05:29.464741 23123 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 85.88
I20250622 20:05:29.464756 23123 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 86 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:29.465001 23360 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250622 20:05:29.466008 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 86 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 84 index: 87. Preceding OpId from leader: term: 86 index: 88. (index mismatch)
I20250622 20:05:29.466217 23360 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250622 20:05:29.567202 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 86 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:29.567305 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 86 FOLLOWER]: Advancing to term 87
I20250622 20:05:29.567988 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.568145 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 87 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:29.568431 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:29.568431 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:29.568720 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 87 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:29.568823 23352 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 87 FOLLOWER]: Leader election lost for term 87. Reason: could not achieve majority
I20250622 20:05:29.570775 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 87 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 86. Current term is 87. Ops: []
I20250622 20:05:29.570983 23363 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 86.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250622 20:05:29.571086 23363 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 86 LEADER]: Stepping down as leader of term 86
I20250622 20:05:29.571122 23363 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 86 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:29.571177 23363 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 86.88, Last appended by leader: 88, Current term: 86, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.571250 23363 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 86 FOLLOWER]: Advancing to term 87
I20250622 20:05:29.665997 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 86 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:29.666100 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 86 FOLLOWER]: Advancing to term 87
I20250622 20:05:29.666800 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.667011 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 87 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:29.667215 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:29.667284 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 87: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:29.667246 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:29.667872 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 87.
I20250622 20:05:29.668072 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 87 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:29.668167 23359 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 87 FOLLOWER]: Leader election won for term 87
I20250622 20:05:29.668233 23359 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 87 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:29.668304 23359 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 86.88, Last appended by leader: 88, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.668735 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 86 to 87, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 87 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:29.688627 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 87 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:29.688747 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 87 FOLLOWER]: Advancing to term 88
I20250622 20:05:29.689400 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.689570 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 88 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:29.689805 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 88 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:29.689815 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 88 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:29.689911 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 87 FOLLOWER]: Advancing to term 88
I20250622 20:05:29.690474 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 88.
I20250622 20:05:29.690658 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 88 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:29.690749 23352 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 88 FOLLOWER]: Leader election won for term 88
I20250622 20:05:29.690810 23352 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 88 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:29.690866 23352 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 86.88, Last appended by leader: 88, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.691335 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 87 to 88, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 88 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:29.786860 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 88 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 86 index: 88. Preceding OpId from leader: term: 88 index: 89. (index mismatch)
I20250622 20:05:29.787210 23352 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250622 20:05:29.788300 23123 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 87 LEADER]: Stepping down as leader of term 87
I20250622 20:05:29.788357 23123 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 87 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:29.788420 23123 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 87.89, Last appended by leader: 89, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.788496 23123 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 87 FOLLOWER]: Advancing to term 88
I20250622 20:05:29.789197 23123 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 88 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 87 index: 89. Preceding OpId from leader: term: 88 index: 89. (term mismatch)
I20250622 20:05:29.789237 23123 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 88
I20250622 20:05:29.789250 23123 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 87.89
I20250622 20:05:29.789263 23123 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 88 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:29.789462 23361 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250622 20:05:29.889011 23362 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 88 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:29.889127 23362 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 88 FOLLOWER]: Advancing to term 89
I20250622 20:05:29.889860 23362 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.890002 23362 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 89 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:29.890231 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 89 candidate_status { last_received { term: 88 index: 89 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:29.890372 23123 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 88 FOLLOWER]: Advancing to term 89
I20250622 20:05:29.890226 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 89 candidate_status { last_received { term: 88 index: 89 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:29.891101 23123 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 89.
I20250622 20:05:29.891194 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 89 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 88. Current term is 89. Ops: []
I20250622 20:05:29.891274 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 89 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:29.891405 23362 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 89 FOLLOWER]: Leader election won for term 89
I20250622 20:05:29.891391 23364 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 88.89, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250622 20:05:29.891498 23362 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 89 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:29.891568 23362 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 88.89, Last appended by leader: 89, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.891610 23352 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 88 LEADER]: Stepping down as leader of term 88
I20250622 20:05:29.891640 23352 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 88 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:29.891687 23352 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 88.89, Last appended by leader: 89, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:29.891749 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 88 FOLLOWER]: Advancing to term 89
I20250622 20:05:29.892017 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 88 to 89, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 89 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:29.993643 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 89 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 88 index: 89. Preceding OpId from leader: term: 89 index: 90. (index mismatch)
I20250622 20:05:29.994007 23362 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250622 20:05:29.995116 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 89 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 88 index: 89. Preceding OpId from leader: term: 89 index: 90. (index mismatch)
I20250622 20:05:29.995314 23362 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20250622 20:05:30.173744 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:30.185482 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 89 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:30.185588 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 89 FOLLOWER]: Advancing to term 90
I20250622 20:05:30.186262 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.186419 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 90 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:30.186690 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 90 candidate_status { last_received { term: 89 index: 90 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:30.186710 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 90 candidate_status { last_received { term: 89 index: 90 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:30.187047 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 90 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:30.187162 23359 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 90 FOLLOWER]: Leader election lost for term 90. Reason: could not achieve majority
I20250622 20:05:30.194010 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 90 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 89. Current term is 90. Ops: []
I20250622 20:05:30.194268 23363 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 89.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250622 20:05:30.194389 23363 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 89 LEADER]: Stepping down as leader of term 89
I20250622 20:05:30.194429 23363 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 89 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:30.194499 23363 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 89.90, Last appended by leader: 90, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.194576 23363 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 89 FOLLOWER]: Advancing to term 90
I20250622 20:05:30.288661 23352 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 89 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:30.288761 23352 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 89 FOLLOWER]: Advancing to term 90
I20250622 20:05:30.289403 23352 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.289546 23352 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 90 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:30.289784 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 90 candidate_status { last_received { term: 89 index: 90 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:30.289784 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 90 candidate_status { last_received { term: 89 index: 90 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:30.289916 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 90 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 90: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:30.290488 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 90.
I20250622 20:05:30.290645 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 90 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:30.290760 23352 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 90 FOLLOWER]: Leader election won for term 90
I20250622 20:05:30.290814 23352 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 90 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:30.290854 23352 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 89.90, Last appended by leader: 90, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.291323 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 89 to 90, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 90 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:30.328773 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 90 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:30.328892 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 90 FOLLOWER]: Advancing to term 91
I20250622 20:05:30.329551 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.329718 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 91 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:30.329977 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 91 candidate_status { last_received { term: 89 index: 90 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:30.329978 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 91 candidate_status { last_received { term: 89 index: 90 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:30.330150 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 90 FOLLOWER]: Advancing to term 91
I20250622 20:05:30.330690 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 91 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 91.
I20250622 20:05:30.330874 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 91 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:30.331013 23359 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 91 FOLLOWER]: Leader election won for term 91
I20250622 20:05:30.331139 23359 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 91 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:30.331199 23359 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 89.90, Last appended by leader: 90, Current term: 91, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.331651 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 90 to 91, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 91 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:30.394987 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 91 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 90. Current term is 91. Ops: []
I20250622 20:05:30.395264 23352 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250622 20:05:30.395385 23352 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 91.91 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 90
I20250622 20:05:30.395421 23364 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 90 LEADER]: Stepping down as leader of term 90
I20250622 20:05:30.395474 23364 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 90 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:30.395551 23364 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 90.91, Last appended by leader: 91, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.395638 23364 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 90 FOLLOWER]: Advancing to term 91
I20250622 20:05:30.433231 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 91 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 90 index: 91. Preceding OpId from leader: term: 91 index: 91. (term mismatch)
I20250622 20:05:30.433317 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 90
I20250622 20:05:30.433346 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 90.91
I20250622 20:05:30.433368 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 91 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:30.433683 23359 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250622 20:05:30.434720 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 91 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 89 index: 90. Preceding OpId from leader: term: 91 index: 91. (index mismatch)
I20250622 20:05:30.434998 23366 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250622 20:05:30.536043 23363 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 91 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:30.536166 23363 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 91 FOLLOWER]: Advancing to term 92
I20250622 20:05:30.536852 23363 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.536991 23363 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 92 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:30.537297 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:30.537297 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:30.537559 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 92 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:30.537637 23363 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 92 FOLLOWER]: Leader election lost for term 92. Reason: could not achieve majority
I20250622 20:05:30.554354 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 92 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 91. Current term is 92. Ops: []
I20250622 20:05:30.554584 23359 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 91.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250622 20:05:30.554697 23359 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 91 LEADER]: Stepping down as leader of term 91
I20250622 20:05:30.554724 23359 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 91 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:30.554771 23359 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 91.91, Last appended by leader: 91, Current term: 91, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.554859 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 91 FOLLOWER]: Advancing to term 92
I20250622 20:05:30.620827 23364 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 91 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:30.620929 23364 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 91 FOLLOWER]: Advancing to term 92
I20250622 20:05:30.621611 23364 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.621803 23364 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 92 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:30.622058 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:30.622045 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:30.622165 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 92: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:30.622766 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 92 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 92.
I20250622 20:05:30.622969 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 92 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:30.623061 23364 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 92 FOLLOWER]: Leader election won for term 92
I20250622 20:05:30.623114 23364 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 92 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:30.623160 23364 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 91.91, Last appended by leader: 91, Current term: 92, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.623628 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 91 to 92, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 92 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:30.682130 23363 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 92 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:30.682248 23363 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 92 FOLLOWER]: Advancing to term 93
I20250622 20:05:30.682960 23363 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.683107 23363 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 93 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:30.683363 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 93 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:30.683363 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 93 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:30.683486 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 92 FOLLOWER]: Advancing to term 93
I20250622 20:05:30.684067 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 93.
I20250622 20:05:30.684255 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 93 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:30.684342 23363 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 93 FOLLOWER]: Leader election won for term 93
I20250622 20:05:30.684409 23363 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 93 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:30.684463 23363 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 91.91, Last appended by leader: 91, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.684872 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 92 to 93, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 93 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:30.715564 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 93 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 92. Current term is 93. Ops: []
I20250622 20:05:30.715966 23364 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250622 20:05:30.716094 23364 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 92 LEADER]: Stepping down as leader of term 92
I20250622 20:05:30.716115 23364 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 92 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:30.716194 23364 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 92.92, Last appended by leader: 92, Current term: 92, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.716267 23364 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 92 FOLLOWER]: Advancing to term 93
I20250622 20:05:30.778378 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 93 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 91 index: 91. Preceding OpId from leader: term: 93 index: 92. (index mismatch)
I20250622 20:05:30.778707 23363 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250622 20:05:30.779747 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 93 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 92 index: 92. Preceding OpId from leader: term: 93 index: 92. (term mismatch)
I20250622 20:05:30.779810 23047 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 91
I20250622 20:05:30.779834 23047 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 92.92
I20250622 20:05:30.779855 23047 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 93 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:30.780009 23363 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250622 20:05:30.966813 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 93 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:30.966935 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 93 FOLLOWER]: Advancing to term 94
I20250622 20:05:30.967589 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.967764 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 94 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:30.968000 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:30.968006 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:30.968264 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:30.968349 23359 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20250622 20:05:30.972657 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 94 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 93. Current term is 94. Ops: []
I20250622 20:05:30.972903 23367 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 93.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250622 20:05:30.973026 23367 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 93 LEADER]: Stepping down as leader of term 93
I20250622 20:05:30.973060 23367 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 93 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:30.973129 23367 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 93.92, Last appended by leader: 92, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.973213 23367 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 93 FOLLOWER]: Advancing to term 94
I20250622 20:05:30.987004 23364 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 93 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:30.987110 23364 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 93 FOLLOWER]: Advancing to term 94
I20250622 20:05:30.987797 23364 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.987977 23364 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 94 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:30.988204 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:30.988211 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:30.988281 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 94: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:30.988873 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 94 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 94.
I20250622 20:05:30.989063 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 94 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:30.989179 23364 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 94 FOLLOWER]: Leader election won for term 94
I20250622 20:05:30.989253 23364 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 94 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:30.989311 23364 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 93.92, Last appended by leader: 92, Current term: 94, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:30.989744 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 93 to 94, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 94 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:31.095348 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 94 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 93 index: 92. Preceding OpId from leader: term: 94 index: 93. (index mismatch)
I20250622 20:05:31.095646 23364 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250622 20:05:31.096766 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 94 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 93 index: 92. Preceding OpId from leader: term: 94 index: 93. (index mismatch)
I20250622 20:05:31.096953 23368 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20250622 20:05:31.174068 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:31.198010 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 94 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:31.198123 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 94 FOLLOWER]: Advancing to term 95
I20250622 20:05:31.198554 23367 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 94 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:31.198628 23367 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 94 FOLLOWER]: Advancing to term 95
I20250622 20:05:31.198823 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.199016 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 95 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:31.199209 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 95 candidate_status { last_received { term: 94 index: 93 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:31.199210 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 95 candidate_status { last_received { term: 94 index: 93 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:31.199414 23367 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.199554 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 95: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:31.199568 23367 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 95 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:31.199682 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 95 candidate_status { last_received { term: 94 index: 93 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:31.199744 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 95 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:31.199785 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 95 candidate_status { last_received { term: 94 index: 93 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:31.199860 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 95: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:31.199888 23359 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 95 FOLLOWER]: Leader election lost for term 95. Reason: could not achieve majority
I20250622 20:05:31.199992 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 95 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:31.200089 23367 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 95 FOLLOWER]: Leader election lost for term 95. Reason: could not achieve majority
I20250622 20:05:31.207031 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 95 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 94. Current term is 95. Ops: []
I20250622 20:05:31.207327 23368 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 94.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250622 20:05:31.207437 23368 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 94 LEADER]: Stepping down as leader of term 94
I20250622 20:05:31.207475 23368 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 94 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:31.207540 23368 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 94.93, Last appended by leader: 93, Current term: 94, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.207633 23368 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 94 FOLLOWER]: Advancing to term 95
I20250622 20:05:31.314414 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 95 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:31.314518 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 95 FOLLOWER]: Advancing to term 96
I20250622 20:05:31.315197 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.315341 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 96 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:31.315548 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 96 candidate_status { last_received { term: 94 index: 93 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:31.315627 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 95 FOLLOWER]: Advancing to term 96
I20250622 20:05:31.315587 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 96 candidate_status { last_received { term: 94 index: 93 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:31.315698 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 95 FOLLOWER]: Advancing to term 96
I20250622 20:05:31.316238 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 96.
I20250622 20:05:31.316299 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 96.
I20250622 20:05:31.316468 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 96 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:31.316562 23359 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 96 FOLLOWER]: Leader election won for term 96
I20250622 20:05:31.316660 23359 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 96 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:31.316701 23359 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 94.93, Last appended by leader: 93, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.317124 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 94 to 96, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 96 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:31.399639 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 96 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 94 index: 93. Preceding OpId from leader: term: 96 index: 94. (index mismatch)
I20250622 20:05:31.399964 23359 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250622 20:05:31.401038 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 96 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 94 index: 93. Preceding OpId from leader: term: 96 index: 94. (index mismatch)
I20250622 20:05:31.401364 23369 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20250622 20:05:31.834568 23368 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 96 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:31.834671 23368 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 96 FOLLOWER]: Advancing to term 97
I20250622 20:05:31.835443 23368 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.835605 23368 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 97 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:31.835836 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 97 candidate_status { last_received { term: 96 index: 94 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:31.835865 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 97 candidate_status { last_received { term: 96 index: 94 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:31.836163 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 97 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:31.836308 23368 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 97 FOLLOWER]: Leader election lost for term 97. Reason: could not achieve majority
I20250622 20:05:31.842307 23362 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 96 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:31.842382 23362 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 96 FOLLOWER]: Advancing to term 97
I20250622 20:05:31.843122 23362 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.843261 23362 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 97 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:31.843485 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 97 candidate_status { last_received { term: 96 index: 94 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:31.843499 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 97 candidate_status { last_received { term: 96 index: 94 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:31.843572 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 97 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 97: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:31.843726 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 97 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:31.843829 23362 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 97 FOLLOWER]: Leader election lost for term 97. Reason: could not achieve majority
I20250622 20:05:31.846547 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 97 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 96. Current term is 97. Ops: []
I20250622 20:05:31.846803 23370 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 96.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250622 20:05:31.847061 23359 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 96 LEADER]: Stepping down as leader of term 96
I20250622 20:05:31.847111 23359 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 96 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:31.847167 23359 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 96.94, Last appended by leader: 94, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.847249 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 96 FOLLOWER]: Advancing to term 97
I20250622 20:05:31.956771 23368 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 97 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:31.956908 23368 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 97 FOLLOWER]: Advancing to term 98
I20250622 20:05:31.957813 23368 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.958105 23368 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 98 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:31.958282 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 98 candidate_status { last_received { term: 96 index: 94 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:31.958297 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 98 candidate_status { last_received { term: 96 index: 94 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:31.958411 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 97 FOLLOWER]: Advancing to term 98
I20250622 20:05:31.958427 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 97 FOLLOWER]: Advancing to term 98
I20250622 20:05:31.959013 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 98.
I20250622 20:05:31.959023 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 98.
I20250622 20:05:31.959208 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 98 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters:
I20250622 20:05:31.959304 23368 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 98 FOLLOWER]: Leader election won for term 98
I20250622 20:05:31.959421 23368 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 98 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:31.959476 23368 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 96.94, Last appended by leader: 94, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:31.959928 22944 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 96 to 98, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 98 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:32.046542 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 98 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 96 index: 94. Preceding OpId from leader: term: 98 index: 95. (index mismatch)
I20250622 20:05:32.046846 23368 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250622 20:05:32.047876 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 98 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 96 index: 94. Preceding OpId from leader: term: 98 index: 95. (index mismatch)
I20250622 20:05:32.048085 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20250622 20:05:32.174296 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:32.235131 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 98 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:32.235234 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 98 FOLLOWER]: Advancing to term 99
I20250622 20:05:32.235900 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.236080 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 99 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:32.236312 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:32.236320 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:32.236557 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:32.236642 23359 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20250622 20:05:32.240944 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 99 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 98. Current term is 99. Ops: []
I20250622 20:05:32.241186 23371 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 98.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250622 20:05:32.241313 23371 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 98 LEADER]: Stepping down as leader of term 98
I20250622 20:05:32.241354 23371 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 98 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:32.241421 23371 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 98.95, Last appended by leader: 95, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.241500 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 98 FOLLOWER]: Advancing to term 99
I20250622 20:05:32.320915 23362 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 98 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:32.321022 23362 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 98 FOLLOWER]: Advancing to term 99
I20250622 20:05:32.321691 23362 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.321856 23362 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 99 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:32.322082 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:32.322129 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:32.322228 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 99: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:32.322770 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 99 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 99.
I20250622 20:05:32.323020 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 99 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:32.323124 23362 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 99 FOLLOWER]: Leader election won for term 99
I20250622 20:05:32.323256 23362 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 99 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:32.323319 23362 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 98.95, Last appended by leader: 95, Current term: 99, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.323765 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 98 to 99, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 99 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:32.361073 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 99 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:32.361176 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 99 FOLLOWER]: Advancing to term 100
I20250622 20:05:32.361835 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.362008 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 100 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:32.362221 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 100 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:32.362226 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 100 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:32.362349 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 99 FOLLOWER]: Advancing to term 100
I20250622 20:05:32.362943 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 100.
I20250622 20:05:32.363206 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 100 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:32.363296 23359 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 100 FOLLOWER]: Leader election won for term 100
I20250622 20:05:32.363410 23359 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 100 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:32.363464 23359 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 98.95, Last appended by leader: 95, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.363932 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 99 to 100, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 100 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:32.410367 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 100 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 99. Current term is 100. Ops: []
I20250622 20:05:32.410662 23362 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250622 20:05:32.410804 23362 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 100.96 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 95
I20250622 20:05:32.410840 23375 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 99 LEADER]: Stepping down as leader of term 99
I20250622 20:05:32.410928 23375 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 99 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:32.411015 23375 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 99.96, Last appended by leader: 96, Current term: 99, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.411111 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 99 FOLLOWER]: Advancing to term 100
I20250622 20:05:32.448474 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 100 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 98 index: 95. Preceding OpId from leader: term: 100 index: 96. (index mismatch)
I20250622 20:05:32.448802 23359 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250622 20:05:32.449759 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 100 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 99 index: 96. Preceding OpId from leader: term: 100 index: 96. (term mismatch)
I20250622 20:05:32.449818 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 95
I20250622 20:05:32.449842 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 99.96
I20250622 20:05:32.449867 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 100 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:32.450044 23376 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250622 20:05:32.747817 23371 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 100 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:32.747915 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 100 FOLLOWER]: Advancing to term 101
I20250622 20:05:32.748637 23371 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.748816 23371 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 101 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:32.749058 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:32.749065 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:32.749099 23375 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 100 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:32.749229 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 100 FOLLOWER]: Advancing to term 101
I20250622 20:05:32.749830 23375 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.749979 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 101 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 101: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:32.749986 23375 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 101 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:32.750161 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:32.750159 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 101 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:32.750181 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:32.750288 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 101 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 101: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:32.750324 23371 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 101 FOLLOWER]: Leader election lost for term 101. Reason: could not achieve majority
I20250622 20:05:32.750388 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 101 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:32.750476 23375 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 101 FOLLOWER]: Leader election lost for term 101. Reason: could not achieve majority
I20250622 20:05:32.752765 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 101 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 100. Current term is 101. Ops: []
I20250622 20:05:32.753015 23376 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 100.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250622 20:05:32.753124 23359 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 100 LEADER]: Stepping down as leader of term 100
I20250622 20:05:32.753183 23359 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 100 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:32.753245 23359 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 100.96, Last appended by leader: 96, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.753319 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 100 FOLLOWER]: Advancing to term 101
I20250622 20:05:32.875777 23371 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 101 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:32.875833 23375 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 101 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:32.875936 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 101 FOLLOWER]: Advancing to term 102
I20250622 20:05:32.875946 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 101 FOLLOWER]: Advancing to term 102
I20250622 20:05:32.876785 23375 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.876785 23371 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.877034 23371 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 102 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:32.877023 23375 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 102 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:32.877226 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 102 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:32.877321 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 102: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:32.877233 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 102 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:32.877233 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 102 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:32.877413 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 102: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:32.877432 23123 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 101 FOLLOWER]: Advancing to term 102
I20250622 20:05:32.878015 23123 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 102.
I20250622 20:05:32.877226 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 102 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:32.878114 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 102: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:32.878185 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 102 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:32.878271 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 102 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:32.878286 23371 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 102 FOLLOWER]: Leader election won for term 102
I20250622 20:05:32.878356 23375 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 102 FOLLOWER]: Leader election lost for term 102. Reason: could not achieve majority
I20250622 20:05:32.878386 23371 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 102 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:32.878523 23371 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 100.96, Last appended by leader: 96, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:32.878974 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 100 to 102, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 102 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:32.956597 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 102 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 100 index: 96. Preceding OpId from leader: term: 102 index: 97. (index mismatch)
I20250622 20:05:32.956907 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250622 20:05:32.957098 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 102 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 100 index: 96. Preceding OpId from leader: term: 102 index: 97. (index mismatch)
I20250622 20:05:32.957296 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20250622 20:05:33.059000 23375 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 102 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:33.059015 23359 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 102 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:33.059128 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 102 FOLLOWER]: Advancing to term 103
I20250622 20:05:33.059139 23359 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 102 FOLLOWER]: Advancing to term 103
I20250622 20:05:33.059953 23359 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.059953 23375 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.060155 23375 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 103 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:33.060173 23359 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 103 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:33.060357 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:33.060384 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:33.060379 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:33.060530 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 103: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:33.060375 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:33.060639 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 103: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:33.060654 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 103 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:33.060736 23359 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 103 FOLLOWER]: Leader election lost for term 103. Reason: could not achieve majority
I20250622 20:05:33.060740 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 103 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:33.060846 23375 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 103 FOLLOWER]: Leader election lost for term 103. Reason: could not achieve majority
I20250622 20:05:33.068964 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 103 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 102. Current term is 103. Ops: []
I20250622 20:05:33.069316 23371 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 102.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250622 20:05:33.069468 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 102 LEADER]: Stepping down as leader of term 102
I20250622 20:05:33.069527 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 102 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:33.069607 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 102.97, Last appended by leader: 97, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.069697 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 102 FOLLOWER]: Advancing to term 103
I20250622 20:05:33.170235 23375 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:33.170344 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 103 FOLLOWER]: Advancing to term 104
I20250622 20:05:33.171061 23375 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.171231 23375 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 104 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:33.171555 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 104 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:33.171651 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 103 FOLLOWER]: Advancing to term 104
I20250622 20:05:33.171557 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 104 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:33.171727 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 103 FOLLOWER]: Advancing to term 104
I20250622 20:05:33.172293 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 104.
I20250622 20:05:33.172341 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 104.
I20250622 20:05:33.172478 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:33.172570 23375 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 104 FOLLOWER]: Leader election won for term 104
I20250622 20:05:33.172693 23375 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 104 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:33.172746 23375 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 102.97, Last appended by leader: 97, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.173218 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 102 to 104, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 104 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:33.174530 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:33.267585 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 104 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 102 index: 97. Preceding OpId from leader: term: 104 index: 98. (index mismatch)
I20250622 20:05:33.267885 23375 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250622 20:05:33.268836 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 104 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 102 index: 97. Preceding OpId from leader: term: 104 index: 98. (index mismatch)
I20250622 20:05:33.269012 23375 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250622 20:05:33.369606 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 104 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:33.369725 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 104 FOLLOWER]: Advancing to term 105
I20250622 20:05:33.370024 23383 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 104 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:33.370105 23383 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 104 FOLLOWER]: Advancing to term 105
I20250622 20:05:33.370484 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.370638 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 105 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:33.370838 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:33.370867 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:33.370864 23383 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.371011 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 105: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:33.371138 23383 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 105 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:33.371181 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:33.371235 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:33.371248 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 105: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:33.371289 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:33.371372 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20250622 20:05:33.371428 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:33.371513 23383 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20250622 20:05:33.376448 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 105 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 104. Current term is 105. Ops: []
I20250622 20:05:33.376704 23375 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 104.98, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250622 20:05:33.376818 23375 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 104 LEADER]: Stepping down as leader of term 104
I20250622 20:05:33.376863 23375 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:33.376919 23375 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 104.98, Last appended by leader: 98, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.376993 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 104 FOLLOWER]: Advancing to term 105
I20250622 20:05:33.475782 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:33.475924 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 105 FOLLOWER]: Advancing to term 106
I20250622 20:05:33.476624 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.476779 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 106 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:33.477020 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 106 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:33.477020 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 106 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:33.477142 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 105 FOLLOWER]: Advancing to term 106
I20250622 20:05:33.477176 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 105 FOLLOWER]: Advancing to term 106
I20250622 20:05:33.477841 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 106 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 106.
I20250622 20:05:33.477841 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 106 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 106.
I20250622 20:05:33.478099 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 106 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters:
I20250622 20:05:33.478214 23378 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 106 FOLLOWER]: Leader election won for term 106
I20250622 20:05:33.478282 23378 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 106 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:33.478340 23378 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 104.98, Last appended by leader: 98, Current term: 106, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.478791 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 104 to 106, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 106 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:33.584481 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 106 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 104 index: 98. Preceding OpId from leader: term: 106 index: 99. (index mismatch)
I20250622 20:05:33.584810 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250622 20:05:33.585810 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 106 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 104 index: 98. Preceding OpId from leader: term: 106 index: 99. (index mismatch)
I20250622 20:05:33.586066 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20250622 20:05:33.686678 23375 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 106 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:33.686789 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 106 FOLLOWER]: Advancing to term 107
I20250622 20:05:33.687485 23375 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.687630 23375 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 107 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:33.687891 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:33.687945 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:33.688151 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 107 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:33.688242 23375 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 107 FOLLOWER]: Leader election lost for term 107. Reason: could not achieve majority
I20250622 20:05:33.698614 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 107 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 106. Current term is 107. Ops: []
I20250622 20:05:33.699016 23371 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 106.99, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250622 20:05:33.699160 23371 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 106 LEADER]: Stepping down as leader of term 106
I20250622 20:05:33.699200 23371 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 106 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:33.699247 23371 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 106.99, Last appended by leader: 99, Current term: 106, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.699307 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 106 FOLLOWER]: Advancing to term 107
I20250622 20:05:33.770697 23383 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 106 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:33.770794 23383 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 106 FOLLOWER]: Advancing to term 107
I20250622 20:05:33.771517 23383 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.771687 23383 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 107 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:33.771931 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:33.771955 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:33.772074 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 107: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:33.772593 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 107.
I20250622 20:05:33.772776 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 107 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:33.772903 23383 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 107 FOLLOWER]: Leader election won for term 107
I20250622 20:05:33.773029 23383 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 107 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:33.773083 23383 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 106.99, Last appended by leader: 99, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.773576 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 106 to 107, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 107 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:33.790725 23375 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 107 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:33.790829 23375 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 107 FOLLOWER]: Advancing to term 108
I20250622 20:05:33.791513 23375 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.791661 23375 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 108 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:33.791908 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 108 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:33.791934 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 108 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:33.792001 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 107 FOLLOWER]: Advancing to term 108
I20250622 20:05:33.792584 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 108 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 108.
I20250622 20:05:33.792845 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 108 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:33.792943 23375 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 108 FOLLOWER]: Leader election won for term 108
I20250622 20:05:33.793011 23375 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 108 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:33.793071 23375 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 106.99, Last appended by leader: 99, Current term: 108, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.793519 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 107 to 108, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 108 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:33.884987 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 108 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 107. Current term is 108. Ops: []
I20250622 20:05:33.885313 23383 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250622 20:05:33.885413 23383 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 107 LEADER]: Stepping down as leader of term 107
I20250622 20:05:33.885432 23383 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 107 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:33.885497 23383 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 107.100, Last appended by leader: 100, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.885576 23383 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 107 FOLLOWER]: Advancing to term 108
I20250622 20:05:33.891948 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 108 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 107 index: 100. Preceding OpId from leader: term: 108 index: 100. (term mismatch)
I20250622 20:05:33.892023 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 99
I20250622 20:05:33.891963 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 108 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 106 index: 99. Preceding OpId from leader: term: 108 index: 100. (index mismatch)
I20250622 20:05:33.892052 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 107.100
I20250622 20:05:33.892174 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:33.892346 23375 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250622 20:05:33.892428 23382 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20250622 20:05:33.994109 23371 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 108 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:33.994218 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 108 FOLLOWER]: Advancing to term 109
I20250622 20:05:33.994930 23371 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.995136 23371 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 109 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:33.995416 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:33.995458 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:33.995537 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 108 FOLLOWER]: Advancing to term 109
I20250622 20:05:33.996112 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 109.
I20250622 20:05:33.996307 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:33.996399 23371 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 109 FOLLOWER]: Leader election won for term 109
I20250622 20:05:33.996467 23371 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 109 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:33.996526 23371 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:33.996968 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 108 to 109, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 109 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:34.017530 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 109 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 108. Current term is 109. Ops: []
I20250622 20:05:34.017856 23382 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 108.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250622 20:05:34.018086 23382 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 109.101 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 100
I20250622 20:05:34.018150 23386 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 108 LEADER]: Stepping down as leader of term 108
I20250622 20:05:34.018198 23386 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 108 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:34.018256 23386 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 108, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.018363 23386 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 108 FOLLOWER]: Advancing to term 109
I20250622 20:05:34.018388 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 109 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 108. Current term is 109. Ops: []
I20250622 20:05:34.080348 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 109 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 108 index: 100. Preceding OpId from leader: term: 109 index: 101. (index mismatch)
I20250622 20:05:34.080647 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250622 20:05:34.081689 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 109 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 108 index: 100. Preceding OpId from leader: term: 109 index: 101. (index mismatch)
I20250622 20:05:34.082095 23384 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20250622 20:05:34.174705 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:34.271037 23386 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 109 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:34.271150 23386 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 109 FOLLOWER]: Advancing to term 110
I20250622 20:05:34.271885 23386 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.272048 23386 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 110 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:34.272286 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 110 candidate_status { last_received { term: 109 index: 101 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:34.272291 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 110 candidate_status { last_received { term: 109 index: 101 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:34.272554 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 110 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:34.272637 23386 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 110 FOLLOWER]: Leader election lost for term 110. Reason: could not achieve majority
I20250622 20:05:34.277415 23383 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 109 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:34.277493 23383 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 109 FOLLOWER]: Advancing to term 110
I20250622 20:05:34.278162 23383 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.278316 23383 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 110 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:34.278509 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 110 candidate_status { last_received { term: 109 index: 101 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:34.278507 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 110 candidate_status { last_received { term: 109 index: 101 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:34.278592 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 110: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:34.278755 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 110 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:34.278857 23383 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 110 FOLLOWER]: Leader election lost for term 110. Reason: could not achieve majority
I20250622 20:05:34.289165 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 110 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 109. Current term is 110. Ops: []
I20250622 20:05:34.289388 23371 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 109.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250622 20:05:34.289494 23371 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 109 LEADER]: Stepping down as leader of term 109
I20250622 20:05:34.289528 23371 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 109 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:34.289572 23371 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 109.101, Last appended by leader: 101, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.289631 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 109 FOLLOWER]: Advancing to term 110
I20250622 20:05:34.391078 23386 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 110 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:34.391196 23386 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 110 FOLLOWER]: Advancing to term 111
I20250622 20:05:34.391891 23386 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.392073 23386 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 111 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:34.392289 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 111 candidate_status { last_received { term: 109 index: 101 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:34.392323 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 111 candidate_status { last_received { term: 109 index: 101 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:34.392374 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 110 FOLLOWER]: Advancing to term 111
I20250622 20:05:34.392377 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 110 FOLLOWER]: Advancing to term 111
I20250622 20:05:34.392953 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 111.
I20250622 20:05:34.392962 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 111.
I20250622 20:05:34.393168 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters:
I20250622 20:05:34.393249 23386 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 111 FOLLOWER]: Leader election won for term 111
I20250622 20:05:34.393307 23386 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 111 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:34.393362 23386 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 109.101, Last appended by leader: 101, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.393790 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 109 to 111, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 111 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:34.483256 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 111 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 109 index: 101. Preceding OpId from leader: term: 111 index: 102. (index mismatch)
I20250622 20:05:34.483533 23386 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250622 20:05:34.484587 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 111 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 109 index: 101. Preceding OpId from leader: term: 111 index: 102. (index mismatch)
I20250622 20:05:34.484820 23386 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20250622 20:05:34.585374 23371 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 111 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:34.585484 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 111 FOLLOWER]: Advancing to term 112
I20250622 20:05:34.586041 23383 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 111 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:34.586122 23383 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 111 FOLLOWER]: Advancing to term 112
I20250622 20:05:34.586269 23371 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.586407 23371 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 112 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:34.586562 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:34.586776 23123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:34.586946 23383 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.586984 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:34.587085 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 112 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 111. Current term is 112. Ops: []
I20250622 20:05:34.587092 23371 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20250622 20:05:34.587222 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:34.587262 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:34.587110 23383 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 112 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:34.587323 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 112: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:34.587357 23382 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 111.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250622 20:05:34.587441 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:34.587504 23382 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 111 LEADER]: Stepping down as leader of term 111
I20250622 20:05:34.587545 23382 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 111 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:34.587612 23382 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 111.102, Last appended by leader: 102, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.587687 23382 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 111 FOLLOWER]: Advancing to term 112
I20250622 20:05:34.588264 23383 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20250622 20:05:34.707728 23371 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 112 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:34.707839 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 112 FOLLOWER]: Advancing to term 113
I20250622 20:05:34.708521 23371 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.708685 23371 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 113 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:34.708912 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 113 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:34.708958 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 113 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:34.709074 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 112 FOLLOWER]: Advancing to term 113
I20250622 20:05:34.708973 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 112 FOLLOWER]: Advancing to term 113
I20250622 20:05:34.709857 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 113.
I20250622 20:05:34.709857 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 113.
I20250622 20:05:34.710139 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:34.710243 23371 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 113 FOLLOWER]: Leader election won for term 113
I20250622 20:05:34.710307 23371 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 113 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:34.710366 23371 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 111.102, Last appended by leader: 102, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.710796 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 111 to 113, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 113 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:34.819383 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 113 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 111 index: 102. Preceding OpId from leader: term: 113 index: 103. (index mismatch)
I20250622 20:05:34.819718 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250622 20:05:34.820771 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 113 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 111 index: 102. Preceding OpId from leader: term: 113 index: 103. (index mismatch)
I20250622 20:05:34.821023 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20250622 20:05:34.921516 23382 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 113 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:34.921623 23382 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 113 FOLLOWER]: Advancing to term 114
I20250622 20:05:34.922034 23383 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 113 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:34.922101 23383 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 113 FOLLOWER]: Advancing to term 114
I20250622 20:05:34.922550 23382 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.922725 23382 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 114 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:34.922870 23383 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.923034 23383 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 114 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:34.923020 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:34.923190 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 114 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 114: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:34.923265 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:34.923275 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:34.923020 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:34.923357 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 114 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 114: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:34.923538 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 114 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:34.923599 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 114 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:34.923642 23383 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 114 FOLLOWER]: Leader election lost for term 114. Reason: could not achieve majority
I20250622 20:05:34.923667 23382 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 114 FOLLOWER]: Leader election lost for term 114. Reason: could not achieve majority
I20250622 20:05:34.934952 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 114 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 113. Current term is 114. Ops: []
I20250622 20:05:34.935231 23378 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 113.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250622 20:05:34.935361 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 113 LEADER]: Stepping down as leader of term 113
I20250622 20:05:34.935390 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 113 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:34.935451 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.103, Last appended by leader: 103, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:34.935547 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 113 FOLLOWER]: Advancing to term 114
I20250622 20:05:35.053135 23382 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 114 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:35.053241 23382 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 114 FOLLOWER]: Advancing to term 115
I20250622 20:05:35.053941 23382 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.054101 23382 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 115 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:35.054363 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 115 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:35.054363 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 115 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:35.054481 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 114 FOLLOWER]: Advancing to term 115
I20250622 20:05:35.054481 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 114 FOLLOWER]: Advancing to term 115
I20250622 20:05:35.055192 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 115.
I20250622 20:05:35.055192 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 115.
I20250622 20:05:35.055430 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:35.055516 23382 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 115 FOLLOWER]: Leader election won for term 115
I20250622 20:05:35.055588 23382 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 115 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:35.055650 23382 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.103, Last appended by leader: 103, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.056203 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 113 to 115, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 115 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:35.146528 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 115 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 113 index: 103. Preceding OpId from leader: term: 115 index: 104. (index mismatch)
I20250622 20:05:35.146813 23382 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250622 20:05:35.147909 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 115 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 113 index: 103. Preceding OpId from leader: term: 115 index: 104. (index mismatch)
I20250622 20:05:35.148191 23387 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20250622 20:05:35.174996 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:35.248164 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 115 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:35.248262 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 115 FOLLOWER]: Advancing to term 116
I20250622 20:05:35.248554 23388 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 115 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:35.248610 23388 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 115 FOLLOWER]: Advancing to term 116
I20250622 20:05:35.249114 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.249305 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 116 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:35.249383 23388 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.249526 23388 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 116 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:35.249511 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:35.249642 23199 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:35.249670 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:35.249668 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 116: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:35.249511 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:35.249742 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 116: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:35.249935 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:35.249979 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:35.250032 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20250622 20:05:35.250079 23388 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20250622 20:05:35.260382 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 116 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 115. Current term is 116. Ops: []
I20250622 20:05:35.260687 23387 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 115.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250622 20:05:35.260812 23387 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 115 LEADER]: Stepping down as leader of term 115
I20250622 20:05:35.260847 23387 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 115 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:35.260931 23387 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 115.104, Last appended by leader: 104, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.261006 23387 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 115 FOLLOWER]: Advancing to term 116
I20250622 20:05:35.384514 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 116 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:35.384624 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 116 FOLLOWER]: Advancing to term 117
I20250622 20:05:35.385382 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.385561 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 117 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:35.385835 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 117 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:35.385864 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 117 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:35.386107 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 116 FOLLOWER]: Advancing to term 117
I20250622 20:05:35.386117 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 116 FOLLOWER]: Advancing to term 117
I20250622 20:05:35.386871 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 117.
I20250622 20:05:35.386878 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 117.
I20250622 20:05:35.387148 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters:
I20250622 20:05:35.387279 23378 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 117 FOLLOWER]: Leader election won for term 117
I20250622 20:05:35.387342 23378 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 117 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:35.387396 23378 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 115.104, Last appended by leader: 104, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.387850 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 115 to 117, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 117 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:35.463307 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 117 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 115 index: 104. Preceding OpId from leader: term: 117 index: 105. (index mismatch)
I20250622 20:05:35.463593 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250622 20:05:35.464620 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 117 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 115 index: 104. Preceding OpId from leader: term: 117 index: 105. (index mismatch)
I20250622 20:05:35.464814 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20250622 20:05:35.565116 23388 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 117 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:35.565233 23388 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 117 FOLLOWER]: Advancing to term 118
I20250622 20:05:35.565773 23387 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 117 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:35.565838 23387 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 117 FOLLOWER]: Advancing to term 118
I20250622 20:05:35.566018 23388 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.566191 23388 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 118 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:35.566352 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:35.566378 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:35.566695 23387 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.566826 23387 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 118 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:35.566844 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 118: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:35.567032 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:35.567056 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:35.567117 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 118: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:35.567174 23388 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I20250622 20:05:35.567237 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:35.567399 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:35.567487 23387 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I20250622 20:05:35.567894 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 118 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 117. Current term is 118. Ops: []
I20250622 20:05:35.568073 23371 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 117.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250622 20:05:35.568153 23371 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 117 LEADER]: Stepping down as leader of term 117
I20250622 20:05:35.568169 23371 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 117 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:35.568213 23371 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 117.105, Last appended by leader: 105, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.568285 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 117 FOLLOWER]: Advancing to term 118
I20250622 20:05:35.667575 23388 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 118 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:35.667686 23388 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 118 FOLLOWER]: Advancing to term 119
I20250622 20:05:35.668411 23388 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.668579 23388 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 119 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:35.668830 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 119 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:35.668828 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 119 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:35.668927 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 118 FOLLOWER]: Advancing to term 119
I20250622 20:05:35.668967 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 118 FOLLOWER]: Advancing to term 119
I20250622 20:05:35.669514 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 119.
I20250622 20:05:35.669525 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 119.
I20250622 20:05:35.669728 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 119 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:35.669843 23388 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 119 FOLLOWER]: Leader election won for term 119
I20250622 20:05:35.669975 23388 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 119 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:35.670028 23388 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 117.105, Last appended by leader: 105, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.670518 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 117 to 119, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 119 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:35.786134 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 119 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 117 index: 105. Preceding OpId from leader: term: 119 index: 106. (index mismatch)
I20250622 20:05:35.786433 23388 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250622 20:05:35.787107 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 119 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 117 index: 105. Preceding OpId from leader: term: 119 index: 106. (index mismatch)
I20250622 20:05:35.787345 23389 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250622 20:05:35.888370 23371 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 119 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:35.888475 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 119 FOLLOWER]: Advancing to term 120
I20250622 20:05:35.889176 23371 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.889353 23371 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 120 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:35.889573 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:35.889570 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:35.889799 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 120 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:35.889889 23371 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 120 FOLLOWER]: Leader election lost for term 120. Reason: could not achieve majority
I20250622 20:05:35.906972 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 120 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 119. Current term is 120. Ops: []
I20250622 20:05:35.907291 23389 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 119.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250622 20:05:35.907424 23389 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 119 LEADER]: Stepping down as leader of term 119
I20250622 20:05:35.907444 23389 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 119 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:35.907508 23389 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 119.106, Last appended by leader: 106, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.907595 23389 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 119 FOLLOWER]: Advancing to term 120
I20250622 20:05:35.979027 23387 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 119 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:35.979121 23387 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 119 FOLLOWER]: Advancing to term 120
I20250622 20:05:35.979790 23387 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.979971 23387 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 120 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:35.980232 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:35.980234 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:35.980365 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 120 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 120: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:35.980991 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 120.
I20250622 20:05:35.981206 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 120 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:35.981302 23387 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 120 FOLLOWER]: Leader election won for term 120
I20250622 20:05:35.981446 23387 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 120 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:35.981509 23387 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 119.106, Last appended by leader: 106, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.981961 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 119 to 120, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 120 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:35.995539 23371 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 120 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:35.995654 23371 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 120 FOLLOWER]: Advancing to term 121
I20250622 20:05:35.996337 23371 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.996515 23371 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 121 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:35.996716 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 121 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:35.996765 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 121 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:35.996842 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 120 FOLLOWER]: Advancing to term 121
I20250622 20:05:35.997437 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 121.
I20250622 20:05:35.997627 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 121 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:35.997712 23371 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 121 FOLLOWER]: Leader election won for term 121
I20250622 20:05:35.997788 23371 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 121 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:35.997844 23371 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 119.106, Last appended by leader: 106, Current term: 121, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:35.998282 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 120 to 121, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 121 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:36.064524 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 121 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 120. Current term is 121. Ops: []
I20250622 20:05:36.064828 23387 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250622 20:05:36.064945 23387 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 121.107 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 106
I20250622 20:05:36.065008 23391 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 120 LEADER]: Stepping down as leader of term 120
I20250622 20:05:36.065052 23391 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 120 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:36.065119 23391 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 120.107, Last appended by leader: 107, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.065209 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 120 FOLLOWER]: Advancing to term 121
I20250622 20:05:36.088502 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 121 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 120 index: 107. Preceding OpId from leader: term: 121 index: 107. (term mismatch)
I20250622 20:05:36.088572 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 121 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 119 index: 106. Preceding OpId from leader: term: 121 index: 107. (index mismatch)
I20250622 20:05:36.088593 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 106
I20250622 20:05:36.088683 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 120.107
I20250622 20:05:36.088712 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 121 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:36.088817 23371 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250622 20:05:36.088940 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20250622 20:05:36.175287 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:36.190778 23389 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 121 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:36.190935 23389 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 121 FOLLOWER]: Advancing to term 122
I20250622 20:05:36.191640 23389 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.191820 23389 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 122 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:36.192059 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:36.192059 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:36.192360 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 122 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:36.192464 23389 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 122 FOLLOWER]: Leader election lost for term 122. Reason: could not achieve majority
I20250622 20:05:36.203145 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 122 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 121. Current term is 122. Ops: []
I20250622 20:05:36.203413 23378 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 121.107, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250622 20:05:36.203536 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 121 LEADER]: Stepping down as leader of term 121
I20250622 20:05:36.203567 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 121 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:36.203629 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 121, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.203712 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 121 FOLLOWER]: Advancing to term 122
I20250622 20:05:36.290647 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 121 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:36.290757 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 121 FOLLOWER]: Advancing to term 122
I20250622 20:05:36.291527 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.291680 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 122 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:36.291916 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:36.291923 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:36.292059 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 122 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 122: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:36.292563 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 122.
I20250622 20:05:36.292722 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:36.292837 23391 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 122 FOLLOWER]: Leader election won for term 122
I20250622 20:05:36.292913 23391 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 122 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:36.293004 23391 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.293437 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 121 to 122, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 122 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:36.327956 23389 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 122 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:36.328061 23389 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 122 FOLLOWER]: Advancing to term 123
I20250622 20:05:36.328723 23389 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.328876 23389 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 123 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:36.329149 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 123 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:36.329149 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 123 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:36.329267 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 122 FOLLOWER]: Advancing to term 123
I20250622 20:05:36.329800 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 123 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 123.
I20250622 20:05:36.330013 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 123 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:36.330096 23389 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 123 FOLLOWER]: Leader election won for term 123
I20250622 20:05:36.330148 23389 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 123 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:36.330204 23389 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 123, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.330686 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 122 to 123, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 123 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:36.405009 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 123 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 122. Current term is 123. Ops: []
I20250622 20:05:36.405323 23391 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250622 20:05:36.405452 23391 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 123.108 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 107
I20250622 20:05:36.405490 23387 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 122 LEADER]: Stepping down as leader of term 122
I20250622 20:05:36.405556 23387 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 122 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:36.405625 23387 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 122.108, Last appended by leader: 108, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.405714 23387 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 122 FOLLOWER]: Advancing to term 123
I20250622 20:05:36.415019 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 123 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 122 index: 108. Preceding OpId from leader: term: 123 index: 108. (term mismatch)
I20250622 20:05:36.415096 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 107
I20250622 20:05:36.415112 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 122.108
I20250622 20:05:36.415127 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 123 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:36.415343 23389 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250622 20:05:36.416366 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 123 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 121 index: 107. Preceding OpId from leader: term: 123 index: 108. (index mismatch)
I20250622 20:05:36.416558 23389 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250622 20:05:36.595690 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 123 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:36.595790 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 123 FOLLOWER]: Advancing to term 124
I20250622 20:05:36.596488 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.596663 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 124 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:36.596906 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:36.596894 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:36.597230 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 124 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:36.597374 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 124 FOLLOWER]: Leader election lost for term 124. Reason: could not achieve majority
I20250622 20:05:36.602078 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 124 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 123. Current term is 124. Ops: []
I20250622 20:05:36.602365 23389 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 123.108, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250622 20:05:36.602494 23389 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 123 LEADER]: Stepping down as leader of term 123
I20250622 20:05:36.602525 23389 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 123 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:36.602602 23389 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 123.108, Last appended by leader: 108, Current term: 123, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.602663 23389 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 123 FOLLOWER]: Advancing to term 124
I20250622 20:05:36.634217 23387 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 123 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:36.634330 23387 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 123 FOLLOWER]: Advancing to term 124
I20250622 20:05:36.635067 23387 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.635241 23387 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 124 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:36.635457 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:36.635524 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 124: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:36.635491 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:36.636173 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 124.
I20250622 20:05:36.636354 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 124 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:36.636446 23387 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 124 FOLLOWER]: Leader election won for term 124
I20250622 20:05:36.636511 23387 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 124 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:36.636571 23387 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 123.108, Last appended by leader: 108, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.637008 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 123 to 124, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 124 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:36.714558 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 124 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 123 index: 108. Preceding OpId from leader: term: 124 index: 109. (index mismatch)
I20250622 20:05:36.714918 23387 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250622 20:05:36.716118 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 124 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 123 index: 108. Preceding OpId from leader: term: 124 index: 109. (index mismatch)
I20250622 20:05:36.716375 23392 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20250622 20:05:36.925702 23389 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 124 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:36.925812 23389 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 124 FOLLOWER]: Advancing to term 125
I20250622 20:05:36.926508 23389 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.926674 23389 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 125 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:36.926918 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:36.926913 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:36.927171 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 125 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:36.927276 23389 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 125 FOLLOWER]: Leader election lost for term 125. Reason: could not achieve majority
I20250622 20:05:36.938107 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 125 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 124. Current term is 125. Ops: []
I20250622 20:05:36.938411 23391 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 124.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250622 20:05:36.938530 23391 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 124 LEADER]: Stepping down as leader of term 124
I20250622 20:05:36.938570 23391 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 124 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:36.938632 23391 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 124.109, Last appended by leader: 109, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.938702 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 124 FOLLOWER]: Advancing to term 125
I20250622 20:05:36.996424 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 124 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:36.996528 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 124 FOLLOWER]: Advancing to term 125
I20250622 20:05:36.997242 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.997411 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 125 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:36.997637 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:36.997654 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:36.997712 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 125: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:36.998340 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 125.
I20250622 20:05:36.998541 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 125 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:36.998636 23378 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 125 FOLLOWER]: Leader election won for term 125
I20250622 20:05:36.998735 23378 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 125 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:36.998777 23378 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 124.109, Last appended by leader: 109, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:36.999243 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 124 to 125, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 125 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:37.028800 23389 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 125 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:37.028918 23389 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 125 FOLLOWER]: Advancing to term 126
I20250622 20:05:37.029614 23389 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.029785 23389 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 126 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:37.030019 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 126 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:37.030023 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 126 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:37.030141 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 125 FOLLOWER]: Advancing to term 126
I20250622 20:05:37.030699 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 126.
I20250622 20:05:37.030948 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:37.031044 23389 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 126 FOLLOWER]: Leader election won for term 126
I20250622 20:05:37.031100 23389 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 126 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:37.031147 23389 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 124.109, Last appended by leader: 109, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.031590 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 125 to 126, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 126 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:37.090585 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 126 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 125. Current term is 126. Ops: []
I20250622 20:05:37.090875 23378 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250622 20:05:37.091019 23378 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 126.110 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 109
I20250622 20:05:37.091073 23393 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 125 LEADER]: Stepping down as leader of term 125
I20250622 20:05:37.091140 23393 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 125 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:37.091215 23393 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.110, Last appended by leader: 110, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.091302 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 125 FOLLOWER]: Advancing to term 126
I20250622 20:05:37.107321 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 126 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 124 index: 109. Preceding OpId from leader: term: 126 index: 110. (index mismatch)
I20250622 20:05:37.107586 23389 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250622 20:05:37.108702 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 126 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 125 index: 110. Preceding OpId from leader: term: 126 index: 110. (term mismatch)
I20250622 20:05:37.108760 23048 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 109
I20250622 20:05:37.108783 23048 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 125.110
I20250622 20:05:37.108801 23048 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 126 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:37.109043 23394 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250622 20:05:37.175539 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:37.209422 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 126 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:37.209535 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 126 FOLLOWER]: Advancing to term 127
I20250622 20:05:37.210250 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.210412 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 127 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:37.210666 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 127 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 126. Current term is 127. Ops: []
I20250622 20:05:37.210657 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:37.210657 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:37.210882 23394 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 126.110, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250622 20:05:37.211012 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 127 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:37.211036 23390 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 126 LEADER]: Stepping down as leader of term 126
I20250622 20:05:37.211071 23390 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 126 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:37.211118 23391 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 127 FOLLOWER]: Leader election lost for term 127. Reason: could not achieve majority
I20250622 20:05:37.211144 23390 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 126.110, Last appended by leader: 110, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.211230 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 126 FOLLOWER]: Advancing to term 127
I20250622 20:05:37.316594 23393 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 126 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:37.316702 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 126 FOLLOWER]: Advancing to term 127
I20250622 20:05:37.317436 23393 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.317603 23393 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 127 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:37.317813 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:37.317813 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:37.317926 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 127 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 127: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:37.318675 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 127.
I20250622 20:05:37.318871 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 127 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:37.319031 23393 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 127 FOLLOWER]: Leader election won for term 127
I20250622 20:05:37.319157 23393 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 127 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:37.319226 23393 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 126.110, Last appended by leader: 110, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.319739 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 126 to 127, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 127 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:37.321439 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 127 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:37.321499 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 127 FOLLOWER]: Advancing to term 128
I20250622 20:05:37.322063 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.322177 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 128 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:37.322381 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 128 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:37.322381 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 128 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:37.322515 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 127 FOLLOWER]: Advancing to term 128
I20250622 20:05:37.323125 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 128 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 128.
I20250622 20:05:37.323310 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 128 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:37.323395 23391 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 128 FOLLOWER]: Leader election won for term 128
I20250622 20:05:37.323448 23391 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 128 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:37.323496 23391 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 126.110, Last appended by leader: 110, Current term: 128, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.323927 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 127 to 128, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 128 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:37.395246 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 128 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 127. Current term is 128. Ops: []
I20250622 20:05:37.395522 23393 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250622 20:05:37.395627 23393 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 127 LEADER]: Stepping down as leader of term 127
I20250622 20:05:37.395646 23393 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 127 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:37.395710 23393 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 127.111, Last appended by leader: 111, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.395784 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 127 FOLLOWER]: Advancing to term 128
I20250622 20:05:37.410599 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 128 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 127 index: 111. Preceding OpId from leader: term: 128 index: 111. (term mismatch)
I20250622 20:05:37.410681 23048 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 110
I20250622 20:05:37.410696 23048 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 127.111
I20250622 20:05:37.410709 23048 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 128 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:37.410996 23391 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250622 20:05:37.412057 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 128 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 126 index: 110. Preceding OpId from leader: term: 128 index: 111. (index mismatch)
I20250622 20:05:37.412269 23395 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250622 20:05:37.595258 23394 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 128 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:37.595379 23394 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 128 FOLLOWER]: Advancing to term 129
I20250622 20:05:37.596125 23394 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.596289 23394 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 129 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:37.596509 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:37.596515 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:37.596786 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 129 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:37.596875 23394 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 129 FOLLOWER]: Leader election lost for term 129. Reason: could not achieve majority
I20250622 20:05:37.613238 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 129 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 128. Current term is 129. Ops: []
I20250622 20:05:37.613519 23391 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 128.111, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250622 20:05:37.613628 23391 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 128 LEADER]: Stepping down as leader of term 128
I20250622 20:05:37.613669 23391 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 128 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:37.613727 23391 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 128.111, Last appended by leader: 111, Current term: 128, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.613787 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 128 FOLLOWER]: Advancing to term 129
I20250622 20:05:37.678514 23393 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 128 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:37.678623 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 128 FOLLOWER]: Advancing to term 129
I20250622 20:05:37.679351 23393 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.679519 23393 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 129 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:37.679746 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:37.679750 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:37.679872 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 129: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:37.680433 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 129.
I20250622 20:05:37.680609 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:37.680709 23393 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 129 FOLLOWER]: Leader election won for term 129
I20250622 20:05:37.680768 23393 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 129 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:37.680830 23393 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 128.111, Last appended by leader: 111, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.681252 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 128 to 129, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 129 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:37.743430 23394 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 129 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:37.743541 23394 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 129 FOLLOWER]: Advancing to term 130
I20250622 20:05:37.744272 23394 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.744449 23394 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 130 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:37.744714 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 130 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:37.744740 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 130 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:37.744839 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 129 FOLLOWER]: Advancing to term 130
I20250622 20:05:37.745468 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 130.
I20250622 20:05:37.745712 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 130 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:37.745831 23394 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 130 FOLLOWER]: Leader election won for term 130
I20250622 20:05:37.745901 23394 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 130 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:37.745961 23394 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 128.111, Last appended by leader: 111, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.746572 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 129 to 130, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 130 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:37.767091 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 130 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 129. Current term is 130. Ops: []
I20250622 20:05:37.767376 23393 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250622 20:05:37.767498 23393 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 129 LEADER]: Stepping down as leader of term 129
I20250622 20:05:37.767536 23393 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 129 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:37.767612 23393 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.112, Last appended by leader: 112, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.767678 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 129 FOLLOWER]: Advancing to term 130
I20250622 20:05:37.821702 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 130 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 128 index: 111. Preceding OpId from leader: term: 130 index: 112. (index mismatch)
I20250622 20:05:37.821991 23394 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250622 20:05:37.823071 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 130 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 129 index: 112. Preceding OpId from leader: term: 130 index: 112. (term mismatch)
I20250622 20:05:37.823128 23048 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 111
I20250622 20:05:37.823149 23048 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 129.112
I20250622 20:05:37.823170 23048 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 130 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:37.823319 23396 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20250622 20:05:37.923570 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 130 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:37.923676 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 130 FOLLOWER]: Advancing to term 131
I20250622 20:05:37.924371 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.924512 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 131 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:37.924775 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 131 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:37.924775 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 131 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:37.924914 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 130 FOLLOWER]: Advancing to term 131
I20250622 20:05:37.925509 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 131.
I20250622 20:05:37.925688 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 131 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:37.925786 23391 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 131 FOLLOWER]: Leader election won for term 131
I20250622 20:05:37.925843 23391 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 131 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:37.925917 23391 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 130.112, Last appended by leader: 112, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.926365 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 130 to 131, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 131 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:37.929553 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 131 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 130. Current term is 131. Ops: []
I20250622 20:05:37.929800 23390 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 130.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250622 20:05:37.929883 23390 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 130 LEADER]: Stepping down as leader of term 130
I20250622 20:05:37.929901 23390 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 130 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:37.929991 23390 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 130.112, Last appended by leader: 112, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:37.930066 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 130 FOLLOWER]: Advancing to term 131
I20250622 20:05:38.033924 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 131 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 130 index: 112. Preceding OpId from leader: term: 131 index: 113. (index mismatch)
I20250622 20:05:38.034241 23391 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250622 20:05:38.035219 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 131 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 130 index: 112. Preceding OpId from leader: term: 131 index: 113. (index mismatch)
I20250622 20:05:38.035418 23397 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250622 20:05:38.136755 23393 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 131 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:38.136870 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 131 FOLLOWER]: Advancing to term 132
I20250622 20:05:38.137565 23393 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.137735 23393 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 132 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:38.137965 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 132 candidate_status { last_received { term: 131 index: 113 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:38.138065 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 131 FOLLOWER]: Advancing to term 132
I20250622 20:05:38.137965 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 132 candidate_status { last_received { term: 131 index: 113 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:38.138630 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 132.
I20250622 20:05:38.138789 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:38.138922 23393 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 132 FOLLOWER]: Leader election won for term 132
I20250622 20:05:38.138998 23393 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 132 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:38.139053 23393 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 131.113, Last appended by leader: 113, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.139530 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 131 to 132, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 132 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:38.143319 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 132 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 131. Current term is 132. Ops: []
I20250622 20:05:38.143631 23391 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 131.113, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250622 20:05:38.143738 23391 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 132.114 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 113
I20250622 20:05:38.143785 23397 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 131 LEADER]: Stepping down as leader of term 131
I20250622 20:05:38.143849 23397 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 131 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:38.143914 23397 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 131.113, Last appended by leader: 113, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.144018 23397 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 131 FOLLOWER]: Advancing to term 132
I20250622 20:05:38.175794 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:38.217049 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 132 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 131 index: 113. Preceding OpId from leader: term: 132 index: 114. (index mismatch)
I20250622 20:05:38.217367 23393 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250622 20:05:38.218475 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 132 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 131 index: 113. Preceding OpId from leader: term: 132 index: 114. (index mismatch)
I20250622 20:05:38.218683 23393 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20250622 20:05:38.488691 23390 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 132 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:38.488807 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 132 FOLLOWER]: Advancing to term 133
I20250622 20:05:38.489518 23390 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.489682 23390 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 133 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:38.489921 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 133 candidate_status { last_received { term: 132 index: 114 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:38.489908 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 133 candidate_status { last_received { term: 132 index: 114 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:38.490190 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:38.490281 23390 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I20250622 20:05:38.490461 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 133 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 132. Current term is 133. Ops: []
I20250622 20:05:38.490645 23378 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 132.114, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250622 20:05:38.490749 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 132 LEADER]: Stepping down as leader of term 132
I20250622 20:05:38.490783 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 132 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:38.490839 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 132.114, Last appended by leader: 114, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.490947 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 132 FOLLOWER]: Advancing to term 133
I20250622 20:05:38.518376 23397 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 132 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:38.518491 23397 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 132 FOLLOWER]: Advancing to term 133
I20250622 20:05:38.519274 23397 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.519428 23397 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 133 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:38.519680 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 133 candidate_status { last_received { term: 132 index: 114 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:38.519757 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 133: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:38.519675 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 133 candidate_status { last_received { term: 132 index: 114 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:38.520383 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 133 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 133.
I20250622 20:05:38.520583 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 133 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:38.520677 23397 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 133 FOLLOWER]: Leader election won for term 133
I20250622 20:05:38.520741 23397 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 133 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:38.520800 23397 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 132.114, Last appended by leader: 114, Current term: 133, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.521267 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 132 to 133, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 133 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:38.609776 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 133 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 132 index: 114. Preceding OpId from leader: term: 133 index: 115. (index mismatch)
I20250622 20:05:38.610088 23397 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250622 20:05:38.611047 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 133 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 132 index: 114. Preceding OpId from leader: term: 133 index: 115. (index mismatch)
I20250622 20:05:38.611303 23391 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250622 20:05:38.711916 23390 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 133 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:38.712026 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 133 FOLLOWER]: Advancing to term 134
I20250622 20:05:38.712734 23390 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.712914 23390 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 134 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:38.713152 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 134 candidate_status { last_received { term: 133 index: 115 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:38.713157 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 134 candidate_status { last_received { term: 133 index: 115 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:38.713455 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 134 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:38.713541 23390 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 134 FOLLOWER]: Leader election lost for term 134. Reason: could not achieve majority
I20250622 20:05:38.733016 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 134 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 133. Current term is 134. Ops: []
I20250622 20:05:38.733345 23391 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 133.115, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250622 20:05:38.733460 23391 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 133 LEADER]: Stepping down as leader of term 133
I20250622 20:05:38.733494 23391 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 133 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:38.733549 23391 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 133.115, Last appended by leader: 115, Current term: 133, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.733621 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 133 FOLLOWER]: Advancing to term 134
I20250622 20:05:38.794939 23393 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 133 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:38.795068 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 133 FOLLOWER]: Advancing to term 134
I20250622 20:05:38.795904 23393 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.796128 23393 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 134 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:38.796386 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 134 candidate_status { last_received { term: 133 index: 115 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:38.796389 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 134 candidate_status { last_received { term: 133 index: 115 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:38.796510 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 134 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 134: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:38.797084 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 134.
I20250622 20:05:38.797259 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 134 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:38.797358 23393 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 134 FOLLOWER]: Leader election won for term 134
I20250622 20:05:38.797413 23393 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 134 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:38.797470 23393 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 133.115, Last appended by leader: 115, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.797917 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 133 to 134, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 134 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:38.820014 23390 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 134 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:38.820122 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 134 FOLLOWER]: Advancing to term 135
I20250622 20:05:38.820792 23390 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.820962 23390 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 135 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:38.821194 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 135 candidate_status { last_received { term: 133 index: 115 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:38.821199 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 135 candidate_status { last_received { term: 133 index: 115 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:38.821311 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 134 FOLLOWER]: Advancing to term 135
I20250622 20:05:38.821877 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 135 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 135.
I20250622 20:05:38.822057 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 135 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:38.822144 23390 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 135 FOLLOWER]: Leader election won for term 135
I20250622 20:05:38.822258 23390 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 135 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:38.822299 23390 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 133.115, Last appended by leader: 115, Current term: 135, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.822734 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 134 to 135, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 135 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:38.889341 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 135 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 134. Current term is 135. Ops: []
I20250622 20:05:38.889689 23393 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250622 20:05:38.889806 23393 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 134 LEADER]: Stepping down as leader of term 134
I20250622 20:05:38.889853 23393 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 134 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:38.889899 23393 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 134.116, Last appended by leader: 116, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:38.889952 23393 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 134 FOLLOWER]: Advancing to term 135
I20250622 20:05:38.903234 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 135 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 133 index: 115. Preceding OpId from leader: term: 135 index: 116. (index mismatch)
I20250622 20:05:38.903509 23390 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250622 20:05:38.904502 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 135 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 134 index: 116. Preceding OpId from leader: term: 135 index: 116. (term mismatch)
I20250622 20:05:38.904553 23048 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 115
I20250622 20:05:38.904577 23048 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 134.116
I20250622 20:05:38.904596 23048 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 135 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:38.904763 23390 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250622 20:05:39.176100 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:39.212319 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 135 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:39.212419 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 135 FOLLOWER]: Advancing to term 136
I20250622 20:05:39.213086 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.213259 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 136 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:39.213485 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:39.213495 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:39.213766 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 136 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:39.213868 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 136 FOLLOWER]: Leader election lost for term 136. Reason: could not achieve majority
I20250622 20:05:39.214133 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 136 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 135. Current term is 136. Ops: []
I20250622 20:05:39.214376 23390 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 135.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250622 20:05:39.214488 23399 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 135 LEADER]: Stepping down as leader of term 135
I20250622 20:05:39.214534 23399 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 135 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:39.214603 23399 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 135.116, Last appended by leader: 116, Current term: 135, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.214721 23399 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 135 FOLLOWER]: Advancing to term 136
I20250622 20:05:39.267113 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 135 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:39.267225 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 135 FOLLOWER]: Advancing to term 136
I20250622 20:05:39.267895 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.268031 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 136 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:39.268285 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:39.268314 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 136 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:39.268373 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 136 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 136: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:39.268874 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 136.
I20250622 20:05:39.269054 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:39.269160 23391 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 136 FOLLOWER]: Leader election won for term 136
I20250622 20:05:39.269289 23391 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 136 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:39.269337 23391 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 135.116, Last appended by leader: 116, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.269743 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 135 to 136, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 136 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:39.340677 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 136 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:39.340785 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 136 FOLLOWER]: Advancing to term 137
I20250622 20:05:39.341486 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.341652 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 137 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:39.341887 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 137 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:39.341887 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 137 candidate_status { last_received { term: 135 index: 116 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:39.342028 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 136 FOLLOWER]: Advancing to term 137
I20250622 20:05:39.342597 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 137 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 137.
I20250622 20:05:39.342813 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 137 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:39.342923 23378 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 137 FOLLOWER]: Leader election won for term 137
I20250622 20:05:39.342993 23378 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 137 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:39.343024 23378 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 135.116, Last appended by leader: 116, Current term: 137, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.343524 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 136 to 137, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 137 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:39.347896 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 137 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 136. Current term is 137. Ops: []
I20250622 20:05:39.348104 23391 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250622 20:05:39.348193 23391 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 136 LEADER]: Stepping down as leader of term 136
I20250622 20:05:39.348234 23391 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 136 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:39.348292 23391 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 136.117, Last appended by leader: 117, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.348358 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 136 FOLLOWER]: Advancing to term 137
I20250622 20:05:39.418829 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 137 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 136 index: 117. Preceding OpId from leader: term: 137 index: 117. (term mismatch)
I20250622 20:05:39.418934 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 116
I20250622 20:05:39.418953 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 136.117
I20250622 20:05:39.418970 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 137 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:39.419209 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250622 20:05:39.420255 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 137 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 135 index: 116. Preceding OpId from leader: term: 137 index: 117. (index mismatch)
I20250622 20:05:39.420464 23401 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20250622 20:05:39.684176 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 137 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:39.684286 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 137 FOLLOWER]: Advancing to term 138
I20250622 20:05:39.684974 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.685168 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 138 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:39.685457 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 138 candidate_status { last_received { term: 137 index: 117 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:39.685492 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 138 candidate_status { last_received { term: 137 index: 117 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:39.685673 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 138 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:39.685778 23391 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 138 FOLLOWER]: Leader election lost for term 138. Reason: could not achieve majority
I20250622 20:05:39.698740 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 138 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 137. Current term is 138. Ops: []
I20250622 20:05:39.699048 23378 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 137.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250622 20:05:39.699182 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 137 LEADER]: Stepping down as leader of term 137
I20250622 20:05:39.699218 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 137 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:39.699281 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 137.117, Last appended by leader: 117, Current term: 137, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.699364 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 137 FOLLOWER]: Advancing to term 138
I20250622 20:05:39.702850 23390 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 137 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:39.702950 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 137 FOLLOWER]: Advancing to term 138
I20250622 20:05:39.703507 23390 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.703631 23390 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 138 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:39.703819 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 138 candidate_status { last_received { term: 137 index: 117 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:39.703830 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 138 candidate_status { last_received { term: 137 index: 117 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:39.703915 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 138: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:39.704424 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 138.
I20250622 20:05:39.704615 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:39.704726 23390 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 138 FOLLOWER]: Leader election won for term 138
I20250622 20:05:39.704778 23390 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 138 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:39.704830 23390 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 137.117, Last appended by leader: 117, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.705299 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 137 to 138, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 138 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:39.782776 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 138 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 137 index: 117. Preceding OpId from leader: term: 138 index: 118. (index mismatch)
I20250622 20:05:39.783082 23390 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250622 20:05:39.784147 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 138 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 137 index: 117. Preceding OpId from leader: term: 138 index: 118. (index mismatch)
I20250622 20:05:39.784428 23402 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250622 20:05:39.884991 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 138 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:39.885097 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 138 FOLLOWER]: Advancing to term 139
I20250622 20:05:39.885792 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.886001 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 139 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:39.886255 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:39.886255 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:39.886510 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:39.886595 23391 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I20250622 20:05:39.907088 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 139 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 138. Current term is 139. Ops: []
I20250622 20:05:39.907367 23390 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 138.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250622 20:05:39.907490 23390 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 138 LEADER]: Stepping down as leader of term 138
I20250622 20:05:39.907526 23390 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 138 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:39.907581 23390 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 138.118, Last appended by leader: 118, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.907656 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 138 FOLLOWER]: Advancing to term 139
I20250622 20:05:39.972719 23401 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 138 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:39.972841 23401 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 138 FOLLOWER]: Advancing to term 139
I20250622 20:05:39.973585 23401 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.973749 23401 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 139 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:39.973968 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:39.973992 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 139 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:39.974083 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 139: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:39.974608 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 139.
I20250622 20:05:39.974794 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:39.974941 23401 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 139 FOLLOWER]: Leader election won for term 139
I20250622 20:05:39.975010 23401 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 139 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:39.975069 23401 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 138.118, Last appended by leader: 118, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:39.975545 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 138 to 139, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 139 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:40.015137 23391 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:40.015265 23391 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 139 FOLLOWER]: Advancing to term 140
I20250622 20:05:40.016224 23391 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.016407 23391 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 140 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:40.016685 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 140 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:40.016762 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 140 candidate_status { last_received { term: 138 index: 118 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:40.016825 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 139 FOLLOWER]: Advancing to term 140
I20250622 20:05:40.017412 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 140 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 140.
I20250622 20:05:40.017601 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 140 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:40.017720 23391 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 140 FOLLOWER]: Leader election won for term 140
I20250622 20:05:40.017884 23391 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 140 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:40.017949 23391 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 138.118, Last appended by leader: 118, Current term: 140, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.018508 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 139 to 140, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 140 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:40.051899 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 140 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 139. Current term is 140. Ops: []
I20250622 20:05:40.052191 23401 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250622 20:05:40.052315 23401 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 139 LEADER]: Stepping down as leader of term 139
I20250622 20:05:40.052356 23401 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 139 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:40.052418 23401 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 139.119, Last appended by leader: 119, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.052493 23401 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 139 FOLLOWER]: Advancing to term 140
I20250622 20:05:40.108994 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 140 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 139 index: 119. Preceding OpId from leader: term: 140 index: 119. (term mismatch)
I20250622 20:05:40.109081 23048 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 118
I20250622 20:05:40.109094 23048 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 139.119
I20250622 20:05:40.109110 23048 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 140 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:40.109364 23391 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250622 20:05:40.110471 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 140 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 138 index: 118. Preceding OpId from leader: term: 140 index: 119. (index mismatch)
I20250622 20:05:40.110702 23391 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250622 20:05:40.176348 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:40.298607 23390 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 140 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:40.298710 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 140 FOLLOWER]: Advancing to term 141
I20250622 20:05:40.299422 23390 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.299575 23390 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 141 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:40.299820 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:40.299826 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:40.300083 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 141 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:40.300179 23390 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 141 FOLLOWER]: Leader election lost for term 141. Reason: could not achieve majority
I20250622 20:05:40.307529 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 141 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 140. Current term is 141. Ops: []
I20250622 20:05:40.307843 23404 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 140.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250622 20:05:40.307979 23404 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 140 LEADER]: Stepping down as leader of term 140
I20250622 20:05:40.308022 23404 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 140 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:40.308087 23404 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 140.119, Last appended by leader: 119, Current term: 140, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.308171 23404 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 140 FOLLOWER]: Advancing to term 141
I20250622 20:05:40.389369 23401 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 140 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:40.389483 23401 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 140 FOLLOWER]: Advancing to term 141
I20250622 20:05:40.390173 23401 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.390360 23401 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 141 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:40.390569 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:40.390583 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 141 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:40.390702 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 141 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 141: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:40.391301 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 141.
I20250622 20:05:40.391479 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 141 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:40.391567 23401 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 141 FOLLOWER]: Leader election won for term 141
I20250622 20:05:40.391628 23401 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 141 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:40.391701 23401 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 140.119, Last appended by leader: 119, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.392138 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 140 to 141, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 141 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:40.439193 23390 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 141 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:40.439306 23390 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 141 FOLLOWER]: Advancing to term 142
I20250622 20:05:40.439985 23390 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.440151 23390 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 142 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:40.440382 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 142 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:40.440382 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 142 candidate_status { last_received { term: 140 index: 119 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:40.440527 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 141 FOLLOWER]: Advancing to term 142
I20250622 20:05:40.441088 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 142.
I20250622 20:05:40.441290 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:40.441376 23390 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 142 FOLLOWER]: Leader election won for term 142
I20250622 20:05:40.441442 23390 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 142 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:40.441499 23390 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 140.119, Last appended by leader: 119, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.441984 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 141 to 142, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 142 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:40.475538 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 142 LEADER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 141. Current term is 142. Ops: []
I20250622 20:05:40.475793 23401 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250622 20:05:40.475901 23401 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer a98c6fe5c70f4075ab88d0df16729372 log is divergent from this leader: its last log entry 142.120 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 119
I20250622 20:05:40.475941 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 141 LEADER]: Stepping down as leader of term 141
I20250622 20:05:40.475984 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 141 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:40.476039 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 141.120, Last appended by leader: 120, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.476145 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 141 FOLLOWER]: Advancing to term 142
I20250622 20:05:40.538491 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 142 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 140 index: 119. Preceding OpId from leader: term: 142 index: 120. (index mismatch)
I20250622 20:05:40.538784 23390 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250622 20:05:40.539832 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 142 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 141 index: 120. Preceding OpId from leader: term: 142 index: 120. (term mismatch)
I20250622 20:05:40.539894 23048 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting all ops after (but not including) 119
I20250622 20:05:40.539919 23048 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: Aborting uncommitted NO_OP operation due to leader change: 141.120
I20250622 20:05:40.539944 23048 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 142 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:40.540110 23390 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20250622 20:05:40.731777 23404 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 142 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:40.731884 23404 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 142 FOLLOWER]: Advancing to term 143
I20250622 20:05:40.732604 23404 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.732775 23404 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 143 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:40.732813 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 142 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:40.732856 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 142 FOLLOWER]: Advancing to term 143
I20250622 20:05:40.733049 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:40.733053 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:40.733572 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.733683 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 143 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 143: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:40.733706 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 143 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:40.733903 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:40.733883 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 143 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:40.733886 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 143 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:40.734001 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 143 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ba5e279945c42bf8ccfa9413268ba24 in current term 143: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:40.734120 23404 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 143 FOLLOWER]: Leader election lost for term 143. Reason: could not achieve majority
I20250622 20:05:40.734222 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 143 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:40.734356 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 143 FOLLOWER]: Leader election lost for term 143. Reason: could not achieve majority
I20250622 20:05:40.737780 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 143 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 142. Current term is 143. Ops: []
I20250622 20:05:40.738073 23405 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 142.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250622 20:05:40.738193 23405 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 142 LEADER]: Stepping down as leader of term 142
I20250622 20:05:40.738227 23405 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 142 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:40.738310 23405 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 142.120, Last appended by leader: 120, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.738400 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 142 FOLLOWER]: Advancing to term 143
I20250622 20:05:40.846520 23404 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 143 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:40.846642 23404 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 143 FOLLOWER]: Advancing to term 144
I20250622 20:05:40.847453 23404 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.847613 23404 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 144 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:40.847852 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 144 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:40.847913 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 143 FOLLOWER]: Advancing to term 144
I20250622 20:05:40.847870 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 144 candidate_status { last_received { term: 142 index: 120 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:40.847966 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 143 FOLLOWER]: Advancing to term 144
I20250622 20:05:40.848482 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 144.
I20250622 20:05:40.848557 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 144.
I20250622 20:05:40.848657 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 144 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:40.848748 23404 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 144 FOLLOWER]: Leader election won for term 144
I20250622 20:05:40.848870 23404 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 144 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:40.848919 23404 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 142.120, Last appended by leader: 120, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:40.849406 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 142 to 144, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 144 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:40.927918 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 144 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 142 index: 120. Preceding OpId from leader: term: 144 index: 121. (index mismatch)
I20250622 20:05:40.928232 23404 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250622 20:05:40.929153 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 144 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 142 index: 120. Preceding OpId from leader: term: 144 index: 121. (index mismatch)
I20250622 20:05:40.929416 23404 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250622 20:05:41.112314 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 144 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:41.112425 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 144 FOLLOWER]: Advancing to term 145
I20250622 20:05:41.113101 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.113283 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 145 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:41.113533 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 145 candidate_status { last_received { term: 144 index: 121 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:41.113536 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 145 candidate_status { last_received { term: 144 index: 121 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:41.113811 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 145 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:41.113901 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 145 FOLLOWER]: Leader election lost for term 145. Reason: could not achieve majority
I20250622 20:05:41.120914 23405 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 144 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:41.120988 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 144 FOLLOWER]: Advancing to term 145
I20250622 20:05:41.121654 23405 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.121806 23405 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 145 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:41.121991 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 145 candidate_status { last_received { term: 144 index: 121 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:41.121997 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 145 candidate_status { last_received { term: 144 index: 121 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:41.122112 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 145: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:41.122252 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 145 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:41.122344 23405 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 145 FOLLOWER]: Leader election lost for term 145. Reason: could not achieve majority
I20250622 20:05:41.123271 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 145 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 144. Current term is 145. Ops: []
I20250622 20:05:41.123461 23404 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 144.121, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250622 20:05:41.123549 23404 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 144 LEADER]: Stepping down as leader of term 144
I20250622 20:05:41.123593 23404 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 144 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:41.123646 23404 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 144.121, Last appended by leader: 121, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.123713 23404 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 144 FOLLOWER]: Advancing to term 145
I20250622 20:05:41.176582 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:41.247736 23405 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 145 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:41.247852 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 145 FOLLOWER]: Advancing to term 146
I20250622 20:05:41.248543 23405 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.248685 23405 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 146 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:41.248905 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 146 candidate_status { last_received { term: 144 index: 121 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:41.248905 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 146 candidate_status { last_received { term: 144 index: 121 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:41.249011 23048 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 145 FOLLOWER]: Advancing to term 146
I20250622 20:05:41.249022 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 145 FOLLOWER]: Advancing to term 146
I20250622 20:05:41.249692 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 146.
I20250622 20:05:41.249692 23048 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 146.
I20250622 20:05:41.249909 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 146 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:41.250038 23405 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 146 FOLLOWER]: Leader election won for term 146
I20250622 20:05:41.250165 23405 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 146 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:41.250243 23405 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 144.121, Last appended by leader: 121, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.250638 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 144 to 146, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 146 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:41.345189 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 146 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 144 index: 121. Preceding OpId from leader: term: 146 index: 122. (index mismatch)
I20250622 20:05:41.345486 23405 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250622 20:05:41.346760 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 146 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 144 index: 121. Preceding OpId from leader: term: 146 index: 122. (index mismatch)
I20250622 20:05:41.346963 23405 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250622 20:05:41.448289 23408 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 146 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:41.448401 23408 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 146 FOLLOWER]: Advancing to term 147
I20250622 20:05:41.449177 23408 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.449445 23408 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 147 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:41.449561 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:41.449604 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:41.449756 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 147 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:41.449812 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 147 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 146. Current term is 147. Ops: []
I20250622 20:05:41.449887 23408 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 147 FOLLOWER]: Leader election lost for term 147. Reason: could not achieve majority
I20250622 20:05:41.450050 23405 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 146.122, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250622 20:05:41.450142 23405 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 146 LEADER]: Stepping down as leader of term 146
I20250622 20:05:41.450181 23405 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 146 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:41.450241 23405 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 146.122, Last appended by leader: 122, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.450315 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 146 FOLLOWER]: Advancing to term 147
I20250622 20:05:41.532212 23404 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 146 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:41.532322 23404 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 146 FOLLOWER]: Advancing to term 147
I20250622 20:05:41.533038 23404 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.533192 23404 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 147 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:41.533433 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:41.533486 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 147 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:41.533509 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 147 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 147: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:41.534145 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 147.
I20250622 20:05:41.534339 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:41.534431 23404 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 147 FOLLOWER]: Leader election won for term 147
I20250622 20:05:41.534495 23404 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 147 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:41.534564 23404 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 146.122, Last appended by leader: 122, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.535030 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 146 to 147, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 147 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:41.566336 23408 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 147 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:41.566443 23408 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 147 FOLLOWER]: Advancing to term 148
I20250622 20:05:41.567157 23408 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.567358 23408 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 148 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:41.567601 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 148 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:41.567718 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 147 FOLLOWER]: Advancing to term 148
I20250622 20:05:41.567601 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 148 candidate_status { last_received { term: 146 index: 122 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:41.568310 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 148.
I20250622 20:05:41.568471 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:41.568560 23408 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 148 FOLLOWER]: Leader election won for term 148
I20250622 20:05:41.568610 23408 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 148 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:41.568650 23408 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 146.122, Last appended by leader: 122, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.569083 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 147 to 148, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 148 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:41.633700 23048 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 148 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 147. Current term is 148. Ops: []
I20250622 20:05:41.634002 23404 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250622 20:05:41.634109 23404 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 148.123 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 122
I20250622 20:05:41.634152 23406 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 147 LEADER]: Stepping down as leader of term 147
I20250622 20:05:41.634193 23406 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 147 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:41.634251 23406 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 147.123, Last appended by leader: 123, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.634336 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 147 FOLLOWER]: Advancing to term 148
I20250622 20:05:41.644451 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 148 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 146 index: 122. Preceding OpId from leader: term: 148 index: 123. (index mismatch)
I20250622 20:05:41.644730 23408 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250622 20:05:41.645619 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 148 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 147 index: 123. Preceding OpId from leader: term: 148 index: 123. (term mismatch)
I20250622 20:05:41.645675 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 122
I20250622 20:05:41.645699 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 147.123
I20250622 20:05:41.645725 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 148 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:41.645884 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250622 20:05:41.746351 23405 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 148 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:41.746465 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 148 FOLLOWER]: Advancing to term 149
I20250622 20:05:41.747231 23405 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.747408 23405 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 149 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:41.747627 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 149 candidate_status { last_received { term: 148 index: 123 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:41.747682 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 148 FOLLOWER]: Advancing to term 149
I20250622 20:05:41.747644 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 149 candidate_status { last_received { term: 148 index: 123 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:41.748292 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 149.
I20250622 20:05:41.748467 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 149 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:41.748584 23405 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 149 FOLLOWER]: Leader election won for term 149
I20250622 20:05:41.748643 23405 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 149 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:41.748724 23405 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 148.123, Last appended by leader: 123, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.749182 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 148 to 149, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 149 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:41.765944 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 149 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 148. Current term is 149. Ops: []
I20250622 20:05:41.766192 23408 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 148.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250622 20:05:41.766297 23408 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 148 LEADER]: Stepping down as leader of term 148
I20250622 20:05:41.766325 23408 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 148 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:41.766377 23408 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 148.123, Last appended by leader: 123, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:41.766461 23408 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 148 FOLLOWER]: Advancing to term 149
I20250622 20:05:41.851656 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 149 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 148 index: 123. Preceding OpId from leader: term: 149 index: 124. (index mismatch)
I20250622 20:05:41.851933 23405 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250622 20:05:41.852928 23048 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 149 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 148 index: 123. Preceding OpId from leader: term: 149 index: 124. (index mismatch)
I20250622 20:05:41.853267 23405 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20250622 20:05:42.030788 23408 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 149 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:42.030920 23408 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 149 FOLLOWER]: Advancing to term 150
I20250622 20:05:42.031608 23408 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.031757 23408 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 150 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:42.031985 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:42.031991 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:42.032272 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 150 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:42.032380 23408 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 150 FOLLOWER]: Leader election lost for term 150. Reason: could not achieve majority
I20250622 20:05:42.034401 23406 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 149 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:42.034440 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 149 FOLLOWER]: Advancing to term 150
I20250622 20:05:42.034979 23406 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.035120 23406 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 150 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:42.035326 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:42.035374 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 150 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:42.035424 23048 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 150 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 150: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:42.035552 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 150 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:42.035642 23406 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 150 FOLLOWER]: Leader election lost for term 150. Reason: could not achieve majority
I20250622 20:05:42.050348 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 150 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 149. Current term is 150. Ops: []
I20250622 20:05:42.050599 23405 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 149.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250622 20:05:42.050724 23405 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 149 LEADER]: Stepping down as leader of term 149
I20250622 20:05:42.050757 23405 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 149 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:42.050818 23405 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.050935 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 149 FOLLOWER]: Advancing to term 150
I20250622 20:05:42.150838 23408 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 150 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:42.150954 23408 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 150 FOLLOWER]: Advancing to term 151
I20250622 20:05:42.151624 23408 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.151790 23408 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 151 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:42.152002 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 151 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:42.152000 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 151 candidate_status { last_received { term: 149 index: 124 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:42.152101 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 150 FOLLOWER]: Advancing to term 151
I20250622 20:05:42.152108 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 150 FOLLOWER]: Advancing to term 151
I20250622 20:05:42.152665 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 151.
I20250622 20:05:42.152680 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 151.
I20250622 20:05:42.152851 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 151 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:42.152944 23408 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 151 FOLLOWER]: Leader election won for term 151
I20250622 20:05:42.153002 23408 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 151 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:42.153064 23408 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 149.124, Last appended by leader: 124, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.153520 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 149 to 151, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 151 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:42.176803 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:42.244025 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 151 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 149 index: 124. Preceding OpId from leader: term: 151 index: 125. (index mismatch)
I20250622 20:05:42.244315 23408 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250622 20:05:42.245528 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 151 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 149 index: 124. Preceding OpId from leader: term: 151 index: 125. (index mismatch)
I20250622 20:05:42.245723 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20250622 20:05:42.346201 23406 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 151 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:42.346352 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 151 FOLLOWER]: Advancing to term 152
I20250622 20:05:42.346496 23405 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 151 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:42.346568 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 151 FOLLOWER]: Advancing to term 152
I20250622 20:05:42.347175 23406 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.347187 23405 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.347398 23406 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 152 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:42.347373 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 152 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 151. Current term is 152. Ops: []
I20250622 20:05:42.347543 23405 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 152 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:42.347633 23378 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 151.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250622 20:05:42.347677 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:42.347681 23048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:42.347726 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:42.347767 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 152 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 152: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:42.347652 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 152 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:42.347893 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 152 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 152: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:42.347738 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 151 LEADER]: Stepping down as leader of term 151
I20250622 20:05:42.347957 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 151 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:42.347986 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 152 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:42.348044 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 151.125, Last appended by leader: 125, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.348086 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 152 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:42.348124 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 151 FOLLOWER]: Advancing to term 152
I20250622 20:05:42.348166 23406 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 152 FOLLOWER]: Leader election lost for term 152. Reason: could not achieve majority
I20250622 20:05:42.348197 23405 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 152 FOLLOWER]: Leader election lost for term 152. Reason: could not achieve majority
I20250622 20:05:42.458743 23405 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 152 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:42.458849 23405 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 152 FOLLOWER]: Advancing to term 153
I20250622 20:05:42.459537 23405 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.459681 23405 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 153 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:42.459909 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 153 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:42.459976 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 152 FOLLOWER]: Advancing to term 153
I20250622 20:05:42.459924 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 153 candidate_status { last_received { term: 151 index: 125 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:42.460050 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 152 FOLLOWER]: Advancing to term 153
I20250622 20:05:42.460529 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 153.
I20250622 20:05:42.460602 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 153.
I20250622 20:05:42.460728 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 153 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters:
I20250622 20:05:42.460824 23405 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 153 FOLLOWER]: Leader election won for term 153
I20250622 20:05:42.460870 23405 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 153 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:42.460927 23405 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 151.125, Last appended by leader: 125, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.461351 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 151 to 153, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 153 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:42.555848 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 153 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 151 index: 125. Preceding OpId from leader: term: 153 index: 126. (index mismatch)
I20250622 20:05:42.556125 23405 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250622 20:05:42.557106 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 153 FOLLOWER]: Refusing update from remote peer a98c6fe5c70f4075ab88d0df16729372: Log matching property violated. Preceding OpId in replica: term: 151 index: 125. Preceding OpId from leader: term: 153 index: 126. (index mismatch)
I20250622 20:05:42.557459 23409 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20250622 20:05:42.658926 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 153 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:42.659060 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 153 FOLLOWER]: Advancing to term 154
I20250622 20:05:42.659759 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.659888 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 154 FOLLOWER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 153. Current term is 154. Ops: []
I20250622 20:05:42.659914 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 154 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:42.660070 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 154 candidate_status { last_received { term: 153 index: 126 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:42.660130 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 154 candidate_status { last_received { term: 153 index: 126 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:42.660154 23411 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 153.126, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250622 20:05:42.660243 23411 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 153 LEADER]: Stepping down as leader of term 153
I20250622 20:05:42.660266 23411 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 153 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:42.660308 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 154 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:42.660341 23411 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 153.126, Last appended by leader: 126, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.660403 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 154 FOLLOWER]: Leader election lost for term 154. Reason: could not achieve majority
I20250622 20:05:42.660429 23411 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 153 FOLLOWER]: Advancing to term 154
I20250622 20:05:42.739117 23406 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 153 FOLLOWER]: Starting leader election (detected failure of leader a98c6fe5c70f4075ab88d0df16729372)
I20250622 20:05:42.739228 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 153 FOLLOWER]: Advancing to term 154
I20250622 20:05:42.739951 23406 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.740150 23406 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 154 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:42.740411 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 154 candidate_status { last_received { term: 153 index: 126 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:42.740408 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 154 candidate_status { last_received { term: 153 index: 126 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:42.740541 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 154 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 154: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:42.741072 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 154.
I20250622 20:05:42.741240 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 154 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:42.741326 23406 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 154 FOLLOWER]: Leader election won for term 154
I20250622 20:05:42.741448 23406 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 154 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:42.741503 23406 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 153.126, Last appended by leader: 126, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.741945 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 153 to 154, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 154 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:42.763725 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 154 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:42.763835 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 154 FOLLOWER]: Advancing to term 155
I20250622 20:05:42.764464 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.764601 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 155 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:42.764827 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 155 candidate_status { last_received { term: 153 index: 126 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:42.764894 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 154 FOLLOWER]: Advancing to term 155
I20250622 20:05:42.764832 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 155 candidate_status { last_received { term: 153 index: 126 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:42.765455 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 155.
I20250622 20:05:42.765632 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 155 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:42.765729 23378 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 155 FOLLOWER]: Leader election won for term 155
I20250622 20:05:42.765786 23378 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 155 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:42.765841 23378 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 153.126, Last appended by leader: 126, Current term: 155, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.766280 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 154 to 155, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 155 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:42.830370 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 155 LEADER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 154. Current term is 155. Ops: []
I20250622 20:05:42.830642 23406 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250622 20:05:42.830732 23406 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 155.127 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 126
I20250622 20:05:42.830775 23412 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 154 LEADER]: Stepping down as leader of term 154
I20250622 20:05:42.830840 23412 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 154 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:42.830935 23412 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 154.127, Last appended by leader: 127, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:42.831049 23412 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 154 FOLLOWER]: Advancing to term 155
I20250622 20:05:42.851063 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 155 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 154 index: 127. Preceding OpId from leader: term: 155 index: 127. (term mismatch)
I20250622 20:05:42.851133 23198 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting all ops after (but not including) 126
I20250622 20:05:42.851150 23198 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: Aborting uncommitted NO_OP operation due to leader change: 154.127
I20250622 20:05:42.851164 23198 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 155 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:42.851382 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250622 20:05:42.852351 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 155 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 153 index: 126. Preceding OpId from leader: term: 155 index: 127. (index mismatch)
I20250622 20:05:42.852627 23413 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20250622 20:05:43.138029 23411 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 155 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:43.138147 23411 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 155 FOLLOWER]: Advancing to term 156
I20250622 20:05:43.138844 23411 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.139043 23411 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 156 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:43.139254 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 156 candidate_status { last_received { term: 155 index: 127 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:43.139268 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 156 candidate_status { last_received { term: 155 index: 127 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:43.139497 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 156 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403
I20250622 20:05:43.139601 23411 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 156 FOLLOWER]: Leader election lost for term 156. Reason: could not achieve majority
I20250622 20:05:43.142479 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 156 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 155. Current term is 156. Ops: []
I20250622 20:05:43.142690 23378 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 155.127, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250622 20:05:43.142786 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 155 LEADER]: Stepping down as leader of term 155
I20250622 20:05:43.142819 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 155 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:43.142881 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 155.127, Last appended by leader: 127, Current term: 155, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.142985 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 155 FOLLOWER]: Advancing to term 156
I20250622 20:05:43.144622 23412 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 155 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:43.144677 23412 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 155 FOLLOWER]: Advancing to term 156
I20250622 20:05:43.145227 23412 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.145361 23412 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 156 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:43.145548 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 156 candidate_status { last_received { term: 155 index: 127 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:43.145552 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 156 candidate_status { last_received { term: 155 index: 127 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:43.145655 23124 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 156 FOLLOWER]: Leader election vote request: Denying vote to candidate 9228796aa80a4254abdbb77b79833403 in current term 156: Already voted for candidate a98c6fe5c70f4075ab88d0df16729372 in this term.
I20250622 20:05:43.146112 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 156.
I20250622 20:05:43.146260 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 156 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:43.146355 23412 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 156 FOLLOWER]: Leader election won for term 156
I20250622 20:05:43.146404 23412 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 156 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:43.146441 23412 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 155.127, Last appended by leader: 127, Current term: 156, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.146853 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 155 to 156, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 156 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:43.177026 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:43.238613 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 156 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 155 index: 127. Preceding OpId from leader: term: 156 index: 128. (index mismatch)
I20250622 20:05:43.238919 23412 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250622 20:05:43.239900 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 156 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 155 index: 127. Preceding OpId from leader: term: 156 index: 128. (index mismatch)
I20250622 20:05:43.240113 23412 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250622 20:05:43.463176 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 156 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:43.463284 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 156 FOLLOWER]: Advancing to term 157
I20250622 20:05:43.463968 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.464128 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 157 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:43.464358 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 157 candidate_status { last_received { term: 156 index: 128 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:43.464378 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 157 candidate_status { last_received { term: 156 index: 128 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:43.464622 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 157 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24; no voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:43.464704 23378 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 157 FOLLOWER]: Leader election lost for term 157. Reason: could not achieve majority
I20250622 20:05:43.484131 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 157 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 156. Current term is 157. Ops: []
I20250622 20:05:43.484395 23406 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 156.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250622 20:05:43.484493 23406 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 156 LEADER]: Stepping down as leader of term 156
I20250622 20:05:43.484509 23406 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 156 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:43.484572 23406 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 156.128, Last appended by leader: 128, Current term: 156, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.484655 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 156 FOLLOWER]: Advancing to term 157
I20250622 20:05:43.514954 23411 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 156 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:43.515057 23411 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 156 FOLLOWER]: Advancing to term 157
I20250622 20:05:43.515719 23411 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.515869 23411 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 157 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:43.516119 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 157 candidate_status { last_received { term: 156 index: 128 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:43.516121 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 157 candidate_status { last_received { term: 156 index: 128 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:43.516250 23047 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 157: Already voted for candidate 3ba5e279945c42bf8ccfa9413268ba24 in this term.
I20250622 20:05:43.516804 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 157.
I20250622 20:05:43.516985 23088 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9228796aa80a4254abdbb77b79833403, a98c6fe5c70f4075ab88d0df16729372; no voters: 3ba5e279945c42bf8ccfa9413268ba24
I20250622 20:05:43.517079 23411 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 157 FOLLOWER]: Leader election won for term 157
I20250622 20:05:43.517232 23411 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 157 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:43.517280 23411 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 156.128, Last appended by leader: 128, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.517741 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 156 to 157, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 157 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:43.579115 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 157 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:43.579214 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 157 FOLLOWER]: Advancing to term 158
I20250622 20:05:43.579860 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.580015 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 158 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:43.580250 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 158 candidate_status { last_received { term: 156 index: 128 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:43.580250 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 158 candidate_status { last_received { term: 156 index: 128 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:43.580404 23198 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 157 FOLLOWER]: Advancing to term 158
I20250622 20:05:43.580922 23198 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 158.
I20250622 20:05:43.581080 23012 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 158 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:43.581197 23378 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 158 FOLLOWER]: Leader election won for term 158
I20250622 20:05:43.581282 23378 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 158 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:43.581346 23378 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 156.128, Last appended by leader: 128, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.581950 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 157 to 158, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 158 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:43.618103 23047 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 158 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 157. Current term is 158. Ops: []
I20250622 20:05:43.618382 23411 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: INVALID_TERM, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250622 20:05:43.618489 23411 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 3ba5e279945c42bf8ccfa9413268ba24 log is divergent from this leader: its last log entry 158.129 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 128
I20250622 20:05:43.618512 23411 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 157 LEADER]: Stepping down as leader of term 157
I20250622 20:05:43.618527 23411 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 157 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:43.618594 23411 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 157.129, Last appended by leader: 129, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.618666 23411 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 157 FOLLOWER]: Advancing to term 158
I20250622 20:05:43.674466 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 158 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 157 index: 129. Preceding OpId from leader: term: 158 index: 129. (term mismatch)
I20250622 20:05:43.674539 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 128
I20250622 20:05:43.674554 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 157.129
I20250622 20:05:43.674568 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 158 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:43.674785 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250622 20:05:43.675998 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 158 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 156 index: 128. Preceding OpId from leader: term: 158 index: 129. (index mismatch)
I20250622 20:05:43.676201 23413 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250622 20:05:43.861203 23406 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 158 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:43.861305 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 158 FOLLOWER]: Advancing to term 159
I20250622 20:05:43.861997 23406 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.862146 23406 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 159 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:43.862372 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 159 candidate_status { last_received { term: 158 index: 129 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:43.862394 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 159 candidate_status { last_received { term: 158 index: 129 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:43.862596 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 159 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9228796aa80a4254abdbb77b79833403; no voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:43.862689 23406 raft_consensus.cc:2747] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 159 FOLLOWER]: Leader election lost for term 159. Reason: could not achieve majority
I20250622 20:05:43.879488 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 159 FOLLOWER]: Rejecting Update request from peer 3ba5e279945c42bf8ccfa9413268ba24 for earlier term 158. Current term is 159. Ops: []
I20250622 20:05:43.879710 23413 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 158.129, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250622 20:05:43.879851 23378 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 158 LEADER]: Stepping down as leader of term 158
I20250622 20:05:43.879894 23378 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 158 LEADER]: Becoming Follower/Learner. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:43.879966 23378 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 158.129, Last appended by leader: 129, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.880067 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 158 FOLLOWER]: Advancing to term 159
I20250622 20:05:43.964556 23411 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 158 FOLLOWER]: Starting leader election (detected failure of leader 3ba5e279945c42bf8ccfa9413268ba24)
I20250622 20:05:43.964655 23411 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 158 FOLLOWER]: Advancing to term 159
I20250622 20:05:43.965298 23411 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.965444 23411 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 159 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591)
I20250622 20:05:43.965667 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 159 candidate_status { last_received { term: 158 index: 129 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:43.965664 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "a98c6fe5c70f4075ab88d0df16729372" candidate_term: 159 candidate_status { last_received { term: 158 index: 129 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:43.965791 23198 raft_consensus.cc:2391] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 159 FOLLOWER]: Leader election vote request: Denying vote to candidate a98c6fe5c70f4075ab88d0df16729372 in current term 159: Already voted for candidate 9228796aa80a4254abdbb77b79833403 in this term.
I20250622 20:05:43.966341 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 159 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98c6fe5c70f4075ab88d0df16729372 in term 159.
I20250622 20:05:43.966521 23089 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [CANDIDATE]: Term 159 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:43.966616 23411 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 159 FOLLOWER]: Leader election won for term 159
I20250622 20:05:43.966681 23411 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 159 LEADER]: Becoming Leader. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:43.966734 23411 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 158.129, Last appended by leader: 129, Current term: 159, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.967180 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 reported cstate change: term changed from 158 to 159, leader changed from 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129) to a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130). New cstate: current_term: 159 leader_uuid: "a98c6fe5c70f4075ab88d0df16729372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:43.986148 23406 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 159 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:43.986236 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 159 FOLLOWER]: Advancing to term 160
I20250622 20:05:43.986867 23406 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.987030 23406 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 160 election: Requested vote from peers 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129:46635), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:43.987254 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 160 candidate_status { last_received { term: 158 index: 129 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:43.987254 23047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "9228796aa80a4254abdbb77b79833403" candidate_term: 160 candidate_status { last_received { term: 158 index: 129 } } ignore_live_leader: false dest_uuid: "3ba5e279945c42bf8ccfa9413268ba24"
I20250622 20:05:43.987407 23047 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 159 FOLLOWER]: Advancing to term 160
I20250622 20:05:43.987952 23047 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9228796aa80a4254abdbb77b79833403 in term 160.
I20250622 20:05:43.988118 23164 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [CANDIDATE]: Term 160 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, 9228796aa80a4254abdbb77b79833403; no voters: a98c6fe5c70f4075ab88d0df16729372
I20250622 20:05:43.988214 23406 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 160 FOLLOWER]: Leader election won for term 160
I20250622 20:05:43.988325 23406 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 160 LEADER]: Becoming Leader. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:43.988377 23406 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 158.129, Last appended by leader: 129, Current term: 160, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:43.988806 22941 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 reported cstate change: term changed from 159 to 160, leader changed from a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130) to 9228796aa80a4254abdbb77b79833403 (127.22.96.131). New cstate: current_term: 160 leader_uuid: "9228796aa80a4254abdbb77b79833403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:44.063958 23198 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 160 LEADER]: Rejecting Update request from peer a98c6fe5c70f4075ab88d0df16729372 for earlier term 159. Current term is 160. Ops: []
I20250622 20:05:44.064256 23411 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250622 20:05:44.064373 23411 consensus_queue.cc:1230] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [LEADER]: Peer 9228796aa80a4254abdbb77b79833403 log is divergent from this leader: its last log entry 160.130 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 129
I20250622 20:05:44.064414 23415 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 159 LEADER]: Stepping down as leader of term 159
I20250622 20:05:44.064460 23415 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 159 LEADER]: Becoming Follower/Learner. State: Replica: a98c6fe5c70f4075ab88d0df16729372, State: Running, Role: LEADER
I20250622 20:05:44.064531 23415 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 159.130, Last appended by leader: 130, Current term: 159, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:44.064617 23415 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 159 FOLLOWER]: Advancing to term 160
I20250622 20:05:44.067153 23047 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 160 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 158 index: 129. Preceding OpId from leader: term: 160 index: 130. (index mismatch)
I20250622 20:05:44.067366 23406 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250622 20:05:44.068215 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 160 FOLLOWER]: Refusing update from remote peer 9228796aa80a4254abdbb77b79833403: Log matching property violated. Preceding OpId in replica: term: 159 index: 130. Preceding OpId from leader: term: 160 index: 130. (term mismatch)
I20250622 20:05:44.068266 23124 pending_rounds.cc:85] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting all ops after (but not including) 129
I20250622 20:05:44.068287 23124 pending_rounds.cc:107] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: Aborting uncommitted NO_OP operation due to leader change: 159.130
I20250622 20:05:44.068310 23124 raft_consensus.cc:2887] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 160 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 20:05:44.068540 23406 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20250622 20:05:44.169203 23378 raft_consensus.cc:491] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 160 FOLLOWER]: Starting leader election (detected failure of leader 9228796aa80a4254abdbb77b79833403)
I20250622 20:05:44.169313 23378 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 160 FOLLOWER]: Advancing to term 161
I20250622 20:05:44.169987 23378 raft_consensus.cc:513] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:44.170156 23378 leader_election.cc:290] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 161 election: Requested vote from peers 9228796aa80a4254abdbb77b79833403 (127.22.96.131:34591), a98c6fe5c70f4075ab88d0df16729372 (127.22.96.130:33869)
I20250622 20:05:44.170377 23124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 161 candidate_status { last_received { term: 160 index: 130 } } ignore_live_leader: false dest_uuid: "a98c6fe5c70f4075ab88d0df16729372"
I20250622 20:05:44.170444 23124 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 160 FOLLOWER]: Advancing to term 161
I20250622 20:05:44.170382 23198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e154e8b9b25c4561a5c344f4dca73dd8" candidate_uuid: "3ba5e279945c42bf8ccfa9413268ba24" candidate_term: 161 candidate_status { last_received { term: 160 index: 130 } } ignore_live_leader: false dest_uuid: "9228796aa80a4254abdbb77b79833403"
I20250622 20:05:44.171000 23124 raft_consensus.cc:2466] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba5e279945c42bf8ccfa9413268ba24 in term 161.
I20250622 20:05:44.171161 23013 leader_election.cc:304] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [CANDIDATE]: Term 161 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ba5e279945c42bf8ccfa9413268ba24, a98c6fe5c70f4075ab88d0df16729372; no voters: 9228796aa80a4254abdbb77b79833403
I20250622 20:05:44.171252 23378 raft_consensus.cc:2802] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 161 FOLLOWER]: Leader election won for term 161
I20250622 20:05:44.171301 23378 raft_consensus.cc:695] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 161 LEADER]: Becoming Leader. State: Replica: 3ba5e279945c42bf8ccfa9413268ba24, State: Running, Role: LEADER
I20250622 20:05:44.171356 23378 consensus_queue.cc:237] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 160.130, Last appended by leader: 130, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:44.171764 22943 catalog_manager.cc:5582] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 reported cstate change: term changed from 160 to 161, leader changed from 9228796aa80a4254abdbb77b79833403 (127.22.96.131) to 3ba5e279945c42bf8ccfa9413268ba24 (127.22.96.129). New cstate: current_term: 161 leader_uuid: "3ba5e279945c42bf8ccfa9413268ba24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:44.174985 23124 raft_consensus.cc:1238] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 161 FOLLOWER]: Rejecting Update request from peer 9228796aa80a4254abdbb77b79833403 for earlier term 160. Current term is 161. Ops: []
I20250622 20:05:44.175191 23406 consensus_queue.cc:1046] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: INVALID_TERM, Last received: 160.130, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20250622 20:05:44.175278 23406 raft_consensus.cc:3053] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 160 LEADER]: Stepping down as leader of term 160
I20250622 20:05:44.175307 23406 raft_consensus.cc:738] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 160 LEADER]: Becoming Follower/Learner. State: Replica: 9228796aa80a4254abdbb77b79833403, State: Running, Role: LEADER
I20250622 20:05:44.175347 23406 consensus_queue.cc:260] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 160.130, Last appended by leader: 130, Current term: 160, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba5e279945c42bf8ccfa9413268ba24" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 46635 } } peers { permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 } } peers { permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 } }
I20250622 20:05:44.175424 23406 raft_consensus.cc:3058] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 160 FOLLOWER]: Advancing to term 161
I20250622 20:05:44.177273 22914 timestamp_advancement-itest.cc:267] GCing logs...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:267: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:332: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250622 20:05:44.177464 22914 tablet_server.cc:178] TabletServer@127.22.96.129:0 shutting down...
I20250622 20:05:44.253428 23124 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 161 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 160 index: 130. Preceding OpId from leader: term: 161 index: 131. (index mismatch)
I20250622 20:05:44.253711 23378 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98c6fe5c70f4075ab88d0df16729372" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20250622 20:05:44.254776 23198 raft_consensus.cc:1273] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 161 FOLLOWER]: Refusing update from remote peer 3ba5e279945c42bf8ccfa9413268ba24: Log matching property violated. Preceding OpId in replica: term: 160 index: 130. Preceding OpId from leader: term: 161 index: 131. (index mismatch)
I20250622 20:05:44.255076 23417 consensus_queue.cc:1035] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9228796aa80a4254abdbb77b79833403" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20250622 20:05:44.379303 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:44.379492 22914 tablet_replica.cc:331] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24: stopping tablet replica
I20250622 20:05:44.379606 22914 raft_consensus.cc:2241] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 161 LEADER]: Raft consensus shutting down.
I20250622 20:05:44.379678 22914 raft_consensus.cc:2270] T e154e8b9b25c4561a5c344f4dca73dd8 P 3ba5e279945c42bf8ccfa9413268ba24 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:44.393033 22914 tablet_server.cc:195] TabletServer@127.22.96.129:0 shutdown complete.
I20250622 20:05:44.394292 22914 tablet_server.cc:178] TabletServer@127.22.96.130:0 shutting down...
I20250622 20:05:44.405748 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:44.405910 22914 tablet_replica.cc:331] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372: stopping tablet replica
I20250622 20:05:44.405982 22914 raft_consensus.cc:2241] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 161 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:44.406044 22914 raft_consensus.cc:2270] T e154e8b9b25c4561a5c344f4dca73dd8 P a98c6fe5c70f4075ab88d0df16729372 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:44.418139 22914 tablet_server.cc:195] TabletServer@127.22.96.130:0 shutdown complete.
I20250622 20:05:44.419066 22914 tablet_server.cc:178] TabletServer@127.22.96.131:0 shutting down...
I20250622 20:05:44.434947 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:44.435083 22914 tablet_replica.cc:331] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403: stopping tablet replica
I20250622 20:05:44.435146 22914 raft_consensus.cc:2241] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 161 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:44.435209 22914 raft_consensus.cc:2270] T e154e8b9b25c4561a5c344f4dca73dd8 P 9228796aa80a4254abdbb77b79833403 [term 161 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:44.447348 22914 tablet_server.cc:195] TabletServer@127.22.96.131:0 shutdown complete.
I20250622 20:05:44.448348 22914 master.cc:561] Master@127.22.96.190:38711 shutting down...
I20250622 20:05:44.576467 22914 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 1 LEADER]: Raft consensus shutting down.
I20250622 20:05:44.576583 22914 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:44.576612 22914 tablet_replica.cc:331] T 00000000000000000000000000000000 P 165c38b7034241a0ba14564d08263eb3: stopping tablet replica
I20250622 20:05:44.588832 22914 master.cc:583] Master@127.22.96.190:38711 shutdown complete.
I20250622 20:05:44.590821 22914 test_util.cc:183] -----------------------------------------------
I20250622 20:05:44.590871 22914 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1750622712029598-22914-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (32557 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20250622 20:05:44.591404 22914 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.96.190:35463
I20250622 20:05:44.591554 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250622 20:05:44.592821 22914 server_base.cc:1048] running on GCE node
W20250622 20:05:44.592933 23421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:44.592907 23419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:44.592913 23418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:44.593226 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:44.593262 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:44.593284 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622744593285 us; error 0 us; skew 500 ppm
I20250622 20:05:44.593855 22914 webserver.cc:469] Webserver started at http://127.22.96.190:42201/ using document root <none> and password file <none>
I20250622 20:05:44.593921 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:44.593968 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:44.594017 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:44.594270 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/master-0-root/instance:
uuid: "ed82a338f227478f8b15c9c542f8994e"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.595155 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:44.595611 23426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.595731 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.595778 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/master-0-root
uuid: "ed82a338f227478f8b15c9c542f8994e"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.595813 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:44.616835 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:44.616986 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:44.619524 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.190:35463
I20250622 20:05:44.621423 23478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.190:35463 every 8 connection(s)
I20250622 20:05:44.621556 23479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:44.622495 23479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e: Bootstrap starting.
I20250622 20:05:44.622737 23479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:44.623195 23479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e: No bootstrap required, opened a new log
I20250622 20:05:44.623297 23479 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed82a338f227478f8b15c9c542f8994e" member_type: VOTER }
I20250622 20:05:44.623340 23479 raft_consensus.cc:383] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:44.623354 23479 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed82a338f227478f8b15c9c542f8994e, State: Initialized, Role: FOLLOWER
I20250622 20:05:44.623390 23479 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed82a338f227478f8b15c9c542f8994e" member_type: VOTER }
I20250622 20:05:44.623436 23479 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:05:44.623458 23479 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:05:44.623477 23479 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:44.623935 23479 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed82a338f227478f8b15c9c542f8994e" member_type: VOTER }
I20250622 20:05:44.623986 23479 leader_election.cc:304] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed82a338f227478f8b15c9c542f8994e; no voters:
I20250622 20:05:44.624064 23479 leader_election.cc:290] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:05:44.624121 23482 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:44.624195 23479 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:05:44.624257 23482 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 1 LEADER]: Becoming Leader. State: Replica: ed82a338f227478f8b15c9c542f8994e, State: Running, Role: LEADER
I20250622 20:05:44.624325 23482 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed82a338f227478f8b15c9c542f8994e" member_type: VOTER }
I20250622 20:05:44.624502 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed82a338f227478f8b15c9c542f8994e. Latest consensus state: current_term: 1 leader_uuid: "ed82a338f227478f8b15c9c542f8994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed82a338f227478f8b15c9c542f8994e" member_type: VOTER } }
I20250622 20:05:44.624595 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:44.624521 23483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed82a338f227478f8b15c9c542f8994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed82a338f227478f8b15c9c542f8994e" member_type: VOTER } }
I20250622 20:05:44.624825 23483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:44.624893 23487 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:05:44.625059 23487 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:05:44.625510 22914 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 20:05:44.625626 23487 catalog_manager.cc:1349] Generated new cluster ID: 6b955bb07f6245b3a3d6892d7f257dc1
I20250622 20:05:44.625674 23487 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:05:44.634935 23487 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:05:44.635351 23487 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:05:44.639359 23487 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e: Generated new TSK 0
I20250622 20:05:44.639425 23487 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:05:44.641175 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:44.642366 23503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:44.642403 22914 server_base.cc:1048] running on GCE node
W20250622 20:05:44.642498 23500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:44.642536 23501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:44.642664 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:44.642704 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:44.642725 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622744642725 us; error 0 us; skew 500 ppm
I20250622 20:05:44.643388 22914 webserver.cc:469] Webserver started at http://127.22.96.129:36059/ using document root <none> and password file <none>
I20250622 20:05:44.643479 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:44.643525 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:44.643577 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:44.643819 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root/instance:
uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.644795 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:44.645351 23508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.645502 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250622 20:05:44.645574 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root
uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.645622 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:44.661928 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:44.662104 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:44.662254 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:44.662449 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:44.662477 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.662513 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:44.662540 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.665282 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.129:33619
I20250622 20:05:44.665345 23571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.129:33619 every 8 connection(s)
I20250622 20:05:44.665644 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:44.666657 23579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:44.666705 23576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:44.666851 23577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:44.667032 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:44.667142 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:44.667176 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:44.667207 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622744667208 us; error 0 us; skew 500 ppm
I20250622 20:05:44.667845 22914 webserver.cc:469] Webserver started at http://127.22.96.130:44269/ using document root <none> and password file <none>
I20250622 20:05:44.667920 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:44.667966 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:44.668015 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:44.668314 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-1-root/instance:
uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.669418 23572 heartbeater.cc:344] Connected to a master server at 127.22.96.190:35463
I20250622 20:05:44.669425 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:44.669509 23572 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:44.669622 23572 heartbeater.cc:507] Master 127.22.96.190:35463 requested a full tablet report, sending...
I20250622 20:05:44.669874 23443 ts_manager.cc:194] Registered new tserver with Master: 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619)
I20250622 20:05:44.669996 23584 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.670106 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.670167 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-1-root
uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.670226 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:44.670634 23443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47256
I20250622 20:05:44.679975 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:44.680186 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:44.680339 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:44.680516 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:44.680542 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.680577 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:44.680608 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.683804 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.130:41405
I20250622 20:05:44.683856 23647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.130:41405 every 8 connection(s)
I20250622 20:05:44.684168 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:44.685253 23652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:44.685343 23651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:44.685432 23654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:44.685881 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:44.685998 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:44.686029 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:44.686046 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622744686047 us; error 0 us; skew 500 ppm
I20250622 20:05:44.686775 22914 webserver.cc:469] Webserver started at http://127.22.96.131:38411/ using document root <none> and password file <none>
I20250622 20:05:44.686863 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:44.686939 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:44.686988 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:44.687278 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-2-root/instance:
uuid: "e328fd66f62c42b7b195e500a1f9d228"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.687436 23648 heartbeater.cc:344] Connected to a master server at 127.22.96.190:35463
I20250622 20:05:44.687492 23648 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:44.687621 23648 heartbeater.cc:507] Master 127.22.96.190:35463 requested a full tablet report, sending...
I20250622 20:05:44.687837 23443 ts_manager.cc:194] Registered new tserver with Master: 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:44.688200 23443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47258
I20250622 20:05:44.688300 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:44.688730 23659 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.688827 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.688870 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-2-root
uuid: "e328fd66f62c42b7b195e500a1f9d228"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:44.688920 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:44.704568 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:44.704808 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:44.704972 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:44.705170 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:44.705205 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.705240 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:44.705276 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.708503 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.131:42235
I20250622 20:05:44.708587 23722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.131:42235 every 8 connection(s)
I20250622 20:05:44.711445 23723 heartbeater.cc:344] Connected to a master server at 127.22.96.190:35463
I20250622 20:05:44.711504 23723 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:44.711602 23723 heartbeater.cc:507] Master 127.22.96.190:35463 requested a full tablet report, sending...
I20250622 20:05:44.711838 23443 ts_manager.cc:194] Registered new tserver with Master: e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235)
I20250622 20:05:44.712060 22914 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003363928s
I20250622 20:05:44.712169 23443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47266
I20250622 20:05:44.718652 22914 test_util.cc:276] Using random seed: -1630163654
I20250622 20:05:44.722705 23443 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47304:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 20:05:44.722838 23443 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:05:44.727258 23536 tablet_service.cc:1468] Processing CreateTablet for tablet 535b87aa6608478b80a8c53904dfc017 (DEFAULT_TABLE table=test-workload [id=cca5e68f0d64421891fc8dc2c19a5ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:44.727407 23536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535b87aa6608478b80a8c53904dfc017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:44.727998 23687 tablet_service.cc:1468] Processing CreateTablet for tablet 535b87aa6608478b80a8c53904dfc017 (DEFAULT_TABLE table=test-workload [id=cca5e68f0d64421891fc8dc2c19a5ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:44.728083 23687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535b87aa6608478b80a8c53904dfc017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:44.728734 23744 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Bootstrap starting.
I20250622 20:05:44.728794 23612 tablet_service.cc:1468] Processing CreateTablet for tablet 535b87aa6608478b80a8c53904dfc017 (DEFAULT_TABLE table=test-workload [id=cca5e68f0d64421891fc8dc2c19a5ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:44.728896 23612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535b87aa6608478b80a8c53904dfc017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:44.729207 23744 tablet_bootstrap.cc:654] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:44.729228 23745 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228: Bootstrap starting.
I20250622 20:05:44.729661 23745 tablet_bootstrap.cc:654] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:44.729789 23744 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: No bootstrap required, opened a new log
I20250622 20:05:44.729843 23744 ts_tablet_manager.cc:1397] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:44.730015 23744 raft_consensus.cc:357] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.730095 23744 raft_consensus.cc:383] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:44.730118 23744 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b4afe55a4b4840afd3c38c97cafe4f, State: Initialized, Role: FOLLOWER
I20250622 20:05:44.730173 23744 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.730309 23748 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3: Bootstrap starting.
I20250622 20:05:44.730335 23744 ts_tablet_manager.cc:1428] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250622 20:05:44.730398 23572 heartbeater.cc:499] Master 127.22.96.190:35463 was elected leader, sending a full tablet report...
I20250622 20:05:44.730736 23745 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228: No bootstrap required, opened a new log
I20250622 20:05:44.730770 23748 tablet_bootstrap.cc:654] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:44.730837 23745 ts_tablet_manager.cc:1397] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250622 20:05:44.731065 23745 raft_consensus.cc:357] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.731132 23745 raft_consensus.cc:383] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:44.731156 23745 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e328fd66f62c42b7b195e500a1f9d228, State: Initialized, Role: FOLLOWER
I20250622 20:05:44.731220 23745 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.731374 23745 ts_tablet_manager.cc:1428] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:44.731400 23723 heartbeater.cc:499] Master 127.22.96.190:35463 was elected leader, sending a full tablet report...
I20250622 20:05:44.731469 23748 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3: No bootstrap required, opened a new log
I20250622 20:05:44.731518 23748 ts_tablet_manager.cc:1397] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:44.731654 23748 raft_consensus.cc:357] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.731715 23748 raft_consensus.cc:383] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:44.731739 23748 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42908b25e09144a3a4bf6f199bfd5aa3, State: Initialized, Role: FOLLOWER
I20250622 20:05:44.731791 23748 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.731961 23748 ts_tablet_manager.cc:1428] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250622 20:05:44.732009 23648 heartbeater.cc:499] Master 127.22.96.190:35463 was elected leader, sending a full tablet report...
I20250622 20:05:44.765607 23749 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:05:44.765709 23749 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.765920 23749 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:44.768663 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" is_pre_election: true
I20250622 20:05:44.768775 23622 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b4afe55a4b4840afd3c38c97cafe4f in term 0.
I20250622 20:05:44.768818 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228" is_pre_election: true
I20250622 20:05:44.768908 23511 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42908b25e09144a3a4bf6f199bfd5aa3, 85b4afe55a4b4840afd3c38c97cafe4f; no voters:
I20250622 20:05:44.768906 23697 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b4afe55a4b4840afd3c38c97cafe4f in term 0.
I20250622 20:05:44.769075 23749 raft_consensus.cc:2802] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:05:44.769127 23749 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:44.769142 23749 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:44.769734 23749 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.769840 23749 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 1 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:44.769969 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:44.770010 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:44.770040 23697 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:44.770087 23622 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:44.770735 23622 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b4afe55a4b4840afd3c38c97cafe4f in term 1.
I20250622 20:05:44.770794 23697 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b4afe55a4b4840afd3c38c97cafe4f in term 1.
I20250622 20:05:44.770924 23511 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42908b25e09144a3a4bf6f199bfd5aa3, 85b4afe55a4b4840afd3c38c97cafe4f; no voters:
I20250622 20:05:44.771004 23749 raft_consensus.cc:2802] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:44.771139 23749 raft_consensus.cc:695] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 1 LEADER]: Becoming Leader. State: Replica: 85b4afe55a4b4840afd3c38c97cafe4f, State: Running, Role: LEADER
I20250622 20:05:44.771201 23749 consensus_queue.cc:237] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:44.771719 23443 catalog_manager.cc:5582] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129). New cstate: current_term: 1 leader_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:44.786509 23622 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 1 FOLLOWER]: Refusing update from remote peer 85b4afe55a4b4840afd3c38c97cafe4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:05:44.786528 23697 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 1 FOLLOWER]: Refusing update from remote peer 85b4afe55a4b4840afd3c38c97cafe4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:05:44.786746 23754 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:05:44.786846 23749 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:05:44.789109 23762 mvcc.cc:204] Tried to move back new op lower bound from 7170550762645221376 to 7170550762583318528. Current Snapshot: MvccSnapshot[applied={T|T < 7170550762644791296}]
I20250622 20:05:45.165601 23751 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85b4afe55a4b4840afd3c38c97cafe4f)
I20250622 20:05:45.165755 23751 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:45.166476 23751 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.166661 23751 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 2 election: Requested vote from peers 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:45.169973 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:45.169977 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:45.170260 23663 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e328fd66f62c42b7b195e500a1f9d228; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, 85b4afe55a4b4840afd3c38c97cafe4f
I20250622 20:05:45.170354 23751 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250622 20:05:45.184407 23752 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85b4afe55a4b4840afd3c38c97cafe4f)
I20250622 20:05:45.184513 23752 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:45.184866 23697 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 2 FOLLOWER]: Rejecting Update request from peer 85b4afe55a4b4840afd3c38c97cafe4f for earlier term 1. Current term is 2. Ops: []
I20250622 20:05:45.185111 23754 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: INVALID_TERM, Last received: 1.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250622 20:05:45.185386 23752 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.185536 23752 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 2 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619)
I20250622 20:05:45.188571 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:45.188596 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:45.188660 23697 raft_consensus.cc:2391] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42908b25e09144a3a4bf6f199bfd5aa3 in current term 2: Already voted for candidate e328fd66f62c42b7b195e500a1f9d228 in this term.
I20250622 20:05:45.188827 23585 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42908b25e09144a3a4bf6f199bfd5aa3; no voters: 85b4afe55a4b4840afd3c38c97cafe4f, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:45.188918 23752 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250622 20:05:45.208173 23622 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 2 FOLLOWER]: Rejecting Update request from peer 85b4afe55a4b4840afd3c38c97cafe4f for earlier term 1. Current term is 2. Ops: []
I20250622 20:05:45.208431 23766 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 }, Status: INVALID_TERM, Last received: 1.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250622 20:05:45.266130 23697 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 2 FOLLOWER]: Rejecting Update request from peer 85b4afe55a4b4840afd3c38c97cafe4f for earlier term 1. Current term is 2. Ops: []
I20250622 20:05:45.266407 23749 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: INVALID_TERM, Last received: 1.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250622 20:05:45.276341 23754 raft_consensus.cc:3053] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 1 LEADER]: Stepping down as leader of term 1
I20250622 20:05:45.276410 23754 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85b4afe55a4b4840afd3c38c97cafe4f, State: Running, Role: LEADER
I20250622 20:05:45.276469 23754 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.276561 23754 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:45.292234 23752 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:45.292326 23752 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:05:45.293045 23752 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.293196 23752 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 3 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619)
I20250622 20:05:45.293414 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 3 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:45.293490 23546 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:05:45.293411 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 3 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:45.293540 23697 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:05:45.294225 23697 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42908b25e09144a3a4bf6f199bfd5aa3 in term 3.
I20250622 20:05:45.294236 23546 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42908b25e09144a3a4bf6f199bfd5aa3 in term 3.
I20250622 20:05:45.294435 23585 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228; no voters:
I20250622 20:05:45.294616 23752 raft_consensus.cc:2802] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 20:05:45.294734 23752 raft_consensus.cc:695] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 3 LEADER]: Becoming Leader. State: Replica: 42908b25e09144a3a4bf6f199bfd5aa3, State: Running, Role: LEADER
I20250622 20:05:45.294812 23752 consensus_queue.cc:237] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.295286 23440 catalog_manager.cc:5582] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 reported cstate change: term changed from 1 to 3, leader changed from 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129) to 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130). New cstate: current_term: 3 leader_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:45.373970 23697 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 3 FOLLOWER]: Refusing update from remote peer 42908b25e09144a3a4bf6f199bfd5aa3: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20250622 20:05:45.374238 23752 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250622 20:05:45.374785 23546 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 3 FOLLOWER]: Refusing update from remote peer 42908b25e09144a3a4bf6f199bfd5aa3: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20250622 20:05:45.375013 23781 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250622 20:05:45.478117 23751 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42908b25e09144a3a4bf6f199bfd5aa3)
I20250622 20:05:45.478210 23751 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:05:45.478919 23751 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.479080 23751 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 4 election: Requested vote from peers 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:45.479328 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:45.479333 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:45.479398 23546 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:05:45.479977 23546 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e328fd66f62c42b7b195e500a1f9d228 in term 4.
I20250622 20:05:45.480144 23663 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f, e328fd66f62c42b7b195e500a1f9d228; no voters: 42908b25e09144a3a4bf6f199bfd5aa3
I20250622 20:05:45.480237 23751 raft_consensus.cc:2802] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 20:05:45.480393 23751 raft_consensus.cc:695] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 4 LEADER]: Becoming Leader. State: Replica: e328fd66f62c42b7b195e500a1f9d228, State: Running, Role: LEADER
I20250622 20:05:45.480472 23751 consensus_queue.cc:237] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.480875 23443 catalog_manager.cc:5582] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 reported cstate change: term changed from 3 to 4, leader changed from 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130) to e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131). New cstate: current_term: 4 leader_uuid: "e328fd66f62c42b7b195e500a1f9d228" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:45.483263 23546 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 4 FOLLOWER]: Rejecting Update request from peer 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 3. Current term is 4. Ops: []
I20250622 20:05:45.483444 23788 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: INVALID_TERM, Last received: 3.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:45.488782 23697 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 4 LEADER]: Rejecting Update request from peer 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 3. Current term is 4. Ops: []
I20250622 20:05:45.488955 23781 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: INVALID_TERM, Last received: 3.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:45.489027 23781 consensus_queue.cc:1230] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer e328fd66f62c42b7b195e500a1f9d228 log is divergent from this leader: its last log entry 4.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20250622 20:05:45.565323 23546 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 4 FOLLOWER]: Rejecting Update request from peer 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 3. Current term is 4. Ops: []
I20250622 20:05:45.565677 23781 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: INVALID_TERM, Last received: 3.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:45.584436 23754 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:45.584530 23754 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 4 FOLLOWER]: Advancing to term 5
I20250622 20:05:45.585204 23754 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.585359 23754 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 5 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:45.585574 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:45.585574 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 5 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:45.585825 23511 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:45.585927 23754 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250622 20:05:45.586241 23622 raft_consensus.cc:3053] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 20:05:45.586277 23622 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42908b25e09144a3a4bf6f199bfd5aa3, State: Running, Role: LEADER
I20250622 20:05:45.586337 23622 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.586422 23622 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:05:45.587023 23622 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 4 FOLLOWER]: Refusing update from remote peer e328fd66f62c42b7b195e500a1f9d228: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 4 index: 19. (index mismatch)
I20250622 20:05:45.587239 23751 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:45.604667 23546 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 5 FOLLOWER]: Rejecting Update request from peer e328fd66f62c42b7b195e500a1f9d228 for earlier term 4. Current term is 5. Ops: []
I20250622 20:05:45.604926 23751 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:45.649549 23546 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 5 FOLLOWER]: Rejecting Update request from peer e328fd66f62c42b7b195e500a1f9d228 for earlier term 4. Current term is 5. Ops: [4.19-4.19]
I20250622 20:05:45.649825 23790 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: INVALID_TERM, Last received: 3.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:45.690244 23754 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:45.690335 23754 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 5 FOLLOWER]: Advancing to term 6
I20250622 20:05:45.691001 23754 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.691164 23754 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 6 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:45.691352 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:45.691362 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:45.691591 23509 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:45.691685 23754 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250622 20:05:45.755565 23546 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 6 FOLLOWER]: Rejecting Update request from peer e328fd66f62c42b7b195e500a1f9d228 for earlier term 4. Current term is 6. Ops: [4.19-4.19]
I20250622 20:05:45.755869 23789 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: INVALID_TERM, Last received: 3.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:45.804533 23751 raft_consensus.cc:3053] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 4 LEADER]: Stepping down as leader of term 4
I20250622 20:05:45.804630 23751 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e328fd66f62c42b7b195e500a1f9d228, State: Running, Role: LEADER
I20250622 20:05:45.804711 23751 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 4.19, Last appended by leader: 19, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.804819 23751 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 4 FOLLOWER]: Advancing to term 5
I20250622 20:05:45.824118 23754 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:45.824223 23754 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 6 FOLLOWER]: Advancing to term 7
I20250622 20:05:45.824915 23754 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.825058 23754 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 7 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:45.825271 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 7 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:45.825271 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 7 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:45.825383 23697 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 5 FOLLOWER]: Advancing to term 7
I20250622 20:05:45.826063 23697 raft_consensus.cc:2408] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 85b4afe55a4b4840afd3c38c97cafe4f for term 7 because replica has last-logged OpId of term: 4 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 18.
I20250622 20:05:45.826242 23509 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:45.826359 23754 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250622 20:05:45.857115 23752 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e328fd66f62c42b7b195e500a1f9d228)
I20250622 20:05:45.857214 23752 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 4 FOLLOWER]: Advancing to term 5
I20250622 20:05:45.857861 23752 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.858024 23752 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 5 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619)
I20250622 20:05:45.858244 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:45.858310 23697 raft_consensus.cc:2366] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 5. Current term is 7.
I20250622 20:05:45.858261 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:45.858359 23546 raft_consensus.cc:2366] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 5. Current term is 7.
I20250622 20:05:45.858469 23585 leader_election.cc:400] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 5 election: Vote denied by peer e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235) with higher term. Message: Invalid argument: T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 5. Current term is 7.
I20250622 20:05:45.858513 23585 leader_election.cc:403] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250622 20:05:45.858556 23588 leader_election.cc:400] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 5 election: Vote denied by peer 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619) with higher term. Message: Invalid argument: T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 5. Current term is 7.
I20250622 20:05:45.858582 23752 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 5 FOLLOWER]: Advancing to term 7
I20250622 20:05:45.859210 23752 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 7 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235) with higher term. Message: Invalid argument: T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 5. Current term is 7.
I20250622 20:05:45.968585 23752 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:45.968679 23752 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 7 FOLLOWER]: Advancing to term 8
I20250622 20:05:45.969311 23752 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.969466 23752 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 8 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619)
I20250622 20:05:45.969686 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 8 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:45.969686 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 8 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:45.969796 23697 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 7 FOLLOWER]: Advancing to term 8
I20250622 20:05:45.969805 23546 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 7 FOLLOWER]: Advancing to term 8
I20250622 20:05:45.970467 23546 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42908b25e09144a3a4bf6f199bfd5aa3 in term 8.
I20250622 20:05:45.970472 23697 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42908b25e09144a3a4bf6f199bfd5aa3 in term 8.
I20250622 20:05:45.970670 23585 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228; no voters:
I20250622 20:05:45.970762 23752 raft_consensus.cc:2802] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 8 FOLLOWER]: Leader election won for term 8
I20250622 20:05:45.970813 23752 raft_consensus.cc:695] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 8 LEADER]: Becoming Leader. State: Replica: 42908b25e09144a3a4bf6f199bfd5aa3, State: Running, Role: LEADER
I20250622 20:05:45.970854 23752 consensus_queue.cc:237] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 4.19, Last appended by leader: 19, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:45.971359 23443 catalog_manager.cc:5582] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 reported cstate change: term changed from 4 to 8, leader changed from e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131) to 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130). New cstate: current_term: 8 leader_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:46.063663 23546 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 8 FOLLOWER]: Refusing update from remote peer 42908b25e09144a3a4bf6f199bfd5aa3: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 8 index: 20. (index mismatch)
I20250622 20:05:46.063934 23752 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:46.068774 23697 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 8 FOLLOWER]: Refusing update from remote peer 42908b25e09144a3a4bf6f199bfd5aa3: Log matching property violated. Preceding OpId in replica: term: 4 index: 19. Preceding OpId from leader: term: 8 index: 20. (index mismatch)
I20250622 20:05:46.068993 23752 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250622 20:05:46.169494 23751 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 42908b25e09144a3a4bf6f199bfd5aa3)
I20250622 20:05:46.169584 23751 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 8 FOLLOWER]: Advancing to term 9
I20250622 20:05:46.170214 23751 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.170360 23751 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 9 election: Requested vote from peers 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:46.170534 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:46.170544 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:46.170794 23662 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e328fd66f62c42b7b195e500a1f9d228; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, 85b4afe55a4b4840afd3c38c97cafe4f
I20250622 20:05:46.170921 23751 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250622 20:05:46.171245 23697 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 9 FOLLOWER]: Rejecting Update request from peer 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 8. Current term is 9. Ops: []
I20250622 20:05:46.171516 23781 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250622 20:05:46.276508 23751 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:46.276605 23751 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 9 FOLLOWER]: Advancing to term 10
I20250622 20:05:46.277256 23751 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.277433 23751 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 10 election: Requested vote from peers 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:46.277611 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 10 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:46.277622 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 10 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:46.277817 23662 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e328fd66f62c42b7b195e500a1f9d228; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, 85b4afe55a4b4840afd3c38c97cafe4f
I20250622 20:05:46.277922 23751 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250622 20:05:46.280382 23697 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Rejecting Update request from peer 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 8. Current term is 10. Ops: []
I20250622 20:05:46.280607 23752 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250622 20:05:46.366806 23754 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 42908b25e09144a3a4bf6f199bfd5aa3)
I20250622 20:05:46.366925 23754 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 8 FOLLOWER]: Advancing to term 9
I20250622 20:05:46.367610 23754 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.367784 23754 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 9 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:46.368009 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:46.368009 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:46.368139 23697 raft_consensus.cc:2366] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85b4afe55a4b4840afd3c38c97cafe4f for earlier term 9. Current term is 10.
I20250622 20:05:46.368285 23509 leader_election.cc:400] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 9 election: Vote denied by peer e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235) with higher term. Message: Invalid argument: T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85b4afe55a4b4840afd3c38c97cafe4f for earlier term 9. Current term is 10.
I20250622 20:05:46.368332 23509 leader_election.cc:403] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20250622 20:05:46.368412 23754 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 9 FOLLOWER]: Advancing to term 10
I20250622 20:05:46.368913 23754 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235) with higher term. Message: Invalid argument: T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 85b4afe55a4b4840afd3c38c97cafe4f for earlier term 9. Current term is 10.
I20250622 20:05:46.376549 23546 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 10 FOLLOWER]: Rejecting Update request from peer 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 8. Current term is 10. Ops: []
I20250622 20:05:46.376797 23788 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250622 20:05:46.385388 23697 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Rejecting Update request from peer 42908b25e09144a3a4bf6f199bfd5aa3 for earlier term 8. Current term is 10. Ops: []
I20250622 20:05:46.385629 23752 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250622 20:05:46.386319 23781 raft_consensus.cc:3053] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 8 LEADER]: Stepping down as leader of term 8
I20250622 20:05:46.386350 23781 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 42908b25e09144a3a4bf6f199bfd5aa3, State: Running, Role: LEADER
I20250622 20:05:46.386409 23781 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.386494 23781 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 8 FOLLOWER]: Advancing to term 9
I20250622 20:05:46.460649 23751 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:46.460752 23751 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 10 FOLLOWER]: Advancing to term 11
I20250622 20:05:46.461416 23751 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.461598 23751 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 11 election: Requested vote from peers 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:46.461799 23546 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 11 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
I20250622 20:05:46.461799 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "e328fd66f62c42b7b195e500a1f9d228" candidate_term: 11 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:46.461896 23546 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 10 FOLLOWER]: Advancing to term 11
I20250622 20:05:46.461900 23622 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 9 FOLLOWER]: Advancing to term 11
I20250622 20:05:46.462582 23546 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e328fd66f62c42b7b195e500a1f9d228 in term 11.
I20250622 20:05:46.462582 23622 raft_consensus.cc:2466] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e328fd66f62c42b7b195e500a1f9d228 in term 11.
I20250622 20:05:46.462792 23663 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f, e328fd66f62c42b7b195e500a1f9d228; no voters:
I20250622 20:05:46.462937 23751 raft_consensus.cc:2802] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 11 FOLLOWER]: Leader election won for term 11
I20250622 20:05:46.463045 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:46.463045 23751 raft_consensus.cc:695] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 11 LEADER]: Becoming Leader. State: Replica: e328fd66f62c42b7b195e500a1f9d228, State: Running, Role: LEADER
I20250622 20:05:46.463124 23751 consensus_queue.cc:237] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.463143 22914 log_reader.cc:385] T 535b87aa6608478b80a8c53904dfc017: removed 1 log segments from log reader
I20250622 20:05:46.463193 22914 log.cc:1079] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Deleting log segment in path: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root/wals/535b87aa6608478b80a8c53904dfc017/wal-000000001 (ops 1-17)
I20250622 20:05:46.463567 22914 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20250622 20:05:46.463627 22914 master.cc:561] Master@127.22.96.190:35463 shutting down...
I20250622 20:05:46.463649 23443 catalog_manager.cc:5582] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 reported cstate change: term changed from 8 to 11, leader changed from 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130) to e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131). New cstate: current_term: 11 leader_uuid: "e328fd66f62c42b7b195e500a1f9d228" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:46.553200 23622 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 11 FOLLOWER]: Refusing update from remote peer e328fd66f62c42b7b195e500a1f9d228: Log matching property violated. Preceding OpId in replica: term: 8 index: 20. Preceding OpId from leader: term: 11 index: 21. (index mismatch)
I20250622 20:05:46.553503 23751 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20250622 20:05:46.554327 23648 heartbeater.cc:646] Failed to heartbeat to 127.22.96.190:35463 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20250622 20:05:46.554587 23546 raft_consensus.cc:1273] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 11 FOLLOWER]: Refusing update from remote peer e328fd66f62c42b7b195e500a1f9d228: Log matching property violated. Preceding OpId in replica: term: 8 index: 20. Preceding OpId from leader: term: 11 index: 21. (index mismatch)
I20250622 20:05:46.554807 23751 consensus_queue.cc:1035] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250622 20:05:46.618685 22914 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 1 LEADER]: Raft consensus shutting down.
I20250622 20:05:46.618818 22914 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:46.618854 22914 tablet_replica.cc:331] T 00000000000000000000000000000000 P ed82a338f227478f8b15c9c542f8994e: stopping tablet replica
I20250622 20:05:46.631067 22914 master.cc:583] Master@127.22.96.190:35463 shutdown complete.
I20250622 20:05:46.632381 22914 tablet_server.cc:178] TabletServer@127.22.96.129:0 shutting down...
W20250622 20:05:46.650419 23663 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250622 20:05:46.650887 23663 consensus_peers.cc:487] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 -> Peer 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619): Couldn't send request to peer 85b4afe55a4b4840afd3c38c97cafe4f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:05:46.654906 23781 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e328fd66f62c42b7b195e500a1f9d228)
I20250622 20:05:46.654986 23781 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 11 FOLLOWER]: Advancing to term 12
I20250622 20:05:46.655288 23754 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 11 FOLLOWER]: Starting leader election (detected failure of leader e328fd66f62c42b7b195e500a1f9d228)
I20250622 20:05:46.655349 23754 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 11 FOLLOWER]: Advancing to term 12
I20250622 20:05:46.655715 23781 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.655858 23781 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 12 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619)
I20250622 20:05:46.655921 23754 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.656018 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:46.656047 23754 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 12 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
I20250622 20:05:46.656270 23697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "e328fd66f62c42b7b195e500a1f9d228"
I20250622 20:05:46.656301 23622 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535b87aa6608478b80a8c53904dfc017" candidate_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "42908b25e09144a3a4bf6f199bfd5aa3"
I20250622 20:05:46.656370 23622 raft_consensus.cc:2391] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 85b4afe55a4b4840afd3c38c97cafe4f in current term 12: Already voted for candidate 42908b25e09144a3a4bf6f199bfd5aa3 in this term.
W20250622 20:05:46.656438 23588 leader_election.cc:336] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 85b4afe55a4b4840afd3c38c97cafe4f (127.22.96.129:33619): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250622 20:05:46.656490 23588 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42908b25e09144a3a4bf6f199bfd5aa3; no voters: 85b4afe55a4b4840afd3c38c97cafe4f, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:46.656520 23511 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:46.656558 23781 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250622 20:05:46.656620 23754 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250622 20:05:46.664443 23622 raft_consensus.cc:1238] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 12 FOLLOWER]: Rejecting Update request from peer e328fd66f62c42b7b195e500a1f9d228 for earlier term 11. Current term is 12. Ops: []
I20250622 20:05:46.664677 23793 consensus_queue.cc:1046] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 }, Status: INVALID_TERM, Last received: 11.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250622 20:05:46.664780 23793 raft_consensus.cc:3053] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 11 LEADER]: Stepping down as leader of term 11
I20250622 20:05:46.664801 23793 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e328fd66f62c42b7b195e500a1f9d228, State: Running, Role: LEADER
I20250622 20:05:46.664870 23793 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.664954 23793 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 11 FOLLOWER]: Advancing to term 12
I20250622 20:05:46.666517 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:46.666621 22914 tablet_replica.cc:331] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: stopping tablet replica
I20250622 20:05:46.666683 22914 raft_consensus.cc:2241] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:46.666728 22914 raft_consensus.cc:2270] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:46.678511 22914 tablet_server.cc:195] TabletServer@127.22.96.129:0 shutdown complete.
I20250622 20:05:46.679417 22914 tablet_server.cc:178] TabletServer@127.22.96.130:0 shutting down...
I20250622 20:05:46.685101 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:46.685256 22914 tablet_replica.cc:331] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3: stopping tablet replica
I20250622 20:05:46.685338 22914 raft_consensus.cc:2241] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 12 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:46.685396 22914 raft_consensus.cc:2270] T 535b87aa6608478b80a8c53904dfc017 P 42908b25e09144a3a4bf6f199bfd5aa3 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:46.697224 22914 tablet_server.cc:195] TabletServer@127.22.96.130:0 shutdown complete.
I20250622 20:05:46.698103 22914 tablet_server.cc:178] TabletServer@127.22.96.131:0 shutting down...
I20250622 20:05:46.709949 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:46.710071 22914 tablet_replica.cc:331] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228: stopping tablet replica
I20250622 20:05:46.710126 22914 raft_consensus.cc:2241] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 12 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:46.710209 22914 raft_consensus.cc:2270] T 535b87aa6608478b80a8c53904dfc017 P e328fd66f62c42b7b195e500a1f9d228 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:46.712049 22914 tablet_server.cc:195] TabletServer@127.22.96.131:0 shutdown complete.
I20250622 20:05:46.712805 22914 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20250622 20:05:46.712939 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:46.714079 23799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:46.714223 23801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:46.714296 22914 server_base.cc:1048] running on GCE node
W20250622 20:05:46.714319 23798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:46.714437 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:46.714478 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:46.714504 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622746714503 us; error 0 us; skew 500 ppm
I20250622 20:05:46.715209 22914 webserver.cc:469] Webserver started at http://127.22.96.129:36059/ using document root <none> and password file <none>
I20250622 20:05:46.715301 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:46.715340 22914 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 20:05:46.715971 22914 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:46.717329 23809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:46.717454 22914 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250622 20:05:46.717509 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root
uuid: "85b4afe55a4b4840afd3c38c97cafe4f"
format_stamp: "Formatted at 2025-06-22 20:05:44 on dist-test-slave-422w"
I20250622 20:05:46.717559 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1750622712029598-22914-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5319
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:46.739307 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:46.739500 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:46.739670 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:46.739954 23816 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250622 20:05:46.740685 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:05:46.740726 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250622 20:05:46.740763 22914 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:05:46.741168 22914 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:05:46.741204 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:46.741207 23816 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Bootstrap starting.
I20250622 20:05:46.742534 23816 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:05:46.742766 23816 tablet_bootstrap.cc:492] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Bootstrap complete.
I20250622 20:05:46.742866 23816 ts_tablet_manager.cc:1397] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250622 20:05:46.743006 23816 raft_consensus.cc:357] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.743067 23816 raft_consensus.cc:738] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b4afe55a4b4840afd3c38c97cafe4f, State: Initialized, Role: FOLLOWER
I20250622 20:05:46.743120 23816 consensus_queue.cc:260] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.743291 23816 ts_tablet_manager.cc:1428] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:46.744428 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.129:33619
I20250622 20:05:46.746443 23875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.129:33619 every 8 connection(s)
W20250622 20:05:46.747341 23876 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250622 20:05:46.752723 22914 tablet_server.cc:178] TabletServer@127.22.96.129:33619 shutting down...
I20250622 20:05:46.853758 23852 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:46.853868 23852 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 12 FOLLOWER]: Advancing to term 13
I20250622 20:05:46.854609 23852 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.854822 23852 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 13 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
W20250622 20:05:46.855459 23810 leader_election.cc:336] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235): Network error: Client connection negotiation failed: client connection to 127.22.96.131:42235: connect: Connection refused (error 111)
W20250622 20:05:46.855544 23812 leader_election.cc:336] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405): Network error: Client connection negotiation failed: client connection to 127.22.96.130:41405: connect: Connection refused (error 111)
I20250622 20:05:46.855616 23812 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:46.855712 23852 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250622 20:05:46.976325 23852 raft_consensus.cc:491] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:46.976423 23852 raft_consensus.cc:3058] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 13 FOLLOWER]: Advancing to term 14
I20250622 20:05:46.977092 23852 raft_consensus.cc:513] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e328fd66f62c42b7b195e500a1f9d228" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 42235 } } peers { permanent_uuid: "85b4afe55a4b4840afd3c38c97cafe4f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 33619 } } peers { permanent_uuid: "42908b25e09144a3a4bf6f199bfd5aa3" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 41405 } }
I20250622 20:05:46.977263 23852 leader_election.cc:290] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 14 election: Requested vote from peers e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235), 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405)
W20250622 20:05:46.977831 23812 leader_election.cc:336] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 42908b25e09144a3a4bf6f199bfd5aa3 (127.22.96.130:41405): Network error: Client connection negotiation failed: client connection to 127.22.96.130:41405: connect: Connection refused (error 111)
W20250622 20:05:46.977895 23810 leader_election.cc:336] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer e328fd66f62c42b7b195e500a1f9d228 (127.22.96.131:42235): Network error: Client connection negotiation failed: client connection to 127.22.96.131:42235: connect: Connection refused (error 111)
I20250622 20:05:46.977936 23810 leader_election.cc:304] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85b4afe55a4b4840afd3c38c97cafe4f; no voters: 42908b25e09144a3a4bf6f199bfd5aa3, e328fd66f62c42b7b195e500a1f9d228
I20250622 20:05:46.978019 23852 raft_consensus.cc:2747] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250622 20:05:46.996919 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:46.997131 22914 tablet_replica.cc:331] stopping tablet replica
I20250622 20:05:46.997195 22914 raft_consensus.cc:2241] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 14 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:46.997251 22914 raft_consensus.cc:2270] T 535b87aa6608478b80a8c53904dfc017 P 85b4afe55a4b4840afd3c38c97cafe4f [term 14 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:47.009006 22914 tablet_server.cc:195] TabletServer@127.22.96.129:33619 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (2422 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20250622 20:05:47.013682 22914 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.96.190:40801
I20250622 20:05:47.013835 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:47.014971 23886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.015065 23883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.015082 23884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:47.015194 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:47.015316 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:47.015352 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:47.015365 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622747015366 us; error 0 us; skew 500 ppm
I20250622 20:05:47.016028 22914 webserver.cc:469] Webserver started at http://127.22.96.190:43715/ using document root <none> and password file <none>
I20250622 20:05:47.016108 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:47.016155 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:47.016207 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:47.016470 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/master-0-root/instance:
uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.017310 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:47.017776 23891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.017896 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.017946 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/master-0-root
uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.017982 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:47.023170 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:47.023329 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:47.025684 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.190:40801
I20250622 20:05:47.026254 23943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.190:40801 every 8 connection(s)
I20250622 20:05:47.026394 23944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:47.027530 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee: Bootstrap starting.
I20250622 20:05:47.027768 23944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:47.028153 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee: No bootstrap required, opened a new log
I20250622 20:05:47.028249 23944 raft_consensus.cc:357] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" member_type: VOTER }
I20250622 20:05:47.028290 23944 raft_consensus.cc:383] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:47.028303 23944 raft_consensus.cc:738] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ea4b1efa084ea1980c2dcb3c9184ee, State: Initialized, Role: FOLLOWER
I20250622 20:05:47.028337 23944 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" member_type: VOTER }
I20250622 20:05:47.028383 23944 raft_consensus.cc:397] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:05:47.028410 23944 raft_consensus.cc:491] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:05:47.028434 23944 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:47.028872 23944 raft_consensus.cc:513] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" member_type: VOTER }
I20250622 20:05:47.028921 23944 leader_election.cc:304] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ea4b1efa084ea1980c2dcb3c9184ee; no voters:
I20250622 20:05:47.029013 23944 leader_election.cc:290] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:05:47.029083 23947 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:47.029176 23944 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:05:47.029191 23947 raft_consensus.cc:695] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 1 LEADER]: Becoming Leader. State: Replica: e0ea4b1efa084ea1980c2dcb3c9184ee, State: Running, Role: LEADER
I20250622 20:05:47.029249 23947 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" member_type: VOTER }
I20250622 20:05:47.029456 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" member_type: VOTER } }
I20250622 20:05:47.029572 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:47.029884 23952 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:05:47.029884 23949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ea4b1efa084ea1980c2dcb3c9184ee. Latest consensus state: current_term: 1 leader_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ea4b1efa084ea1980c2dcb3c9184ee" member_type: VOTER } }
I20250622 20:05:47.029996 23949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:47.030083 23952 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:05:47.030544 22914 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 20:05:47.030666 23952 catalog_manager.cc:1349] Generated new cluster ID: 7a20dcfe220c46689953a536ef423412
I20250622 20:05:47.030717 23952 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:05:47.046555 23952 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:05:47.047030 23952 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:05:47.059048 23952 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee: Generated new TSK 0
I20250622 20:05:47.059128 23952 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:05:47.062398 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:47.063709 23966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.063777 23965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.063733 23968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:47.063961 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:47.064050 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:47.064083 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:47.064101 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622747064102 us; error 0 us; skew 500 ppm
I20250622 20:05:47.064709 22914 webserver.cc:469] Webserver started at http://127.22.96.129:39005/ using document root <none> and password file <none>
I20250622 20:05:47.064795 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:47.064833 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:47.064869 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:47.065099 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root/instance:
uuid: "06d0e411419c4639ab8da0c41cc2b04f"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.065917 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:47.066300 23973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.066427 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.066480 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root
uuid: "06d0e411419c4639ab8da0c41cc2b04f"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.066516 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:47.112183 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:47.112397 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:47.112538 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:47.112735 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:47.112766 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.112789 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:47.112818 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.115657 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.129:45439
I20250622 20:05:47.115712 24036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.129:45439 every 8 connection(s)
I20250622 20:05:47.116043 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:47.117216 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.117233 24041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.117353 24044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:47.117523 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:47.117627 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:47.117659 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:47.117681 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622747117681 us; error 0 us; skew 500 ppm
I20250622 20:05:47.118382 22914 webserver.cc:469] Webserver started at http://127.22.96.130:42649/ using document root <none> and password file <none>
I20250622 20:05:47.118471 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:47.118522 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:47.118602 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:47.118863 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-1-root/instance:
uuid: "7ca07ef4cd96472a8604113c07a20380"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.119853 24037 heartbeater.cc:344] Connected to a master server at 127.22.96.190:40801
I20250622 20:05:47.119915 24037 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:47.119979 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:47.120020 24037 heartbeater.cc:507] Master 127.22.96.190:40801 requested a full tablet report, sending...
I20250622 20:05:47.120263 23908 ts_manager.cc:194] Registered new tserver with Master: 06d0e411419c4639ab8da0c41cc2b04f (127.22.96.129:45439)
I20250622 20:05:47.120517 24049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.120649 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.120697 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-1-root
uuid: "7ca07ef4cd96472a8604113c07a20380"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.120738 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:47.120996 23908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51970
I20250622 20:05:47.125396 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:47.125587 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:47.125725 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:47.125905 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:47.125932 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.125962 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:47.125990 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.129109 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.130:38379
I20250622 20:05:47.129172 24112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.130:38379 every 8 connection(s)
I20250622 20:05:47.129472 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:47.130766 24116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.130772 24117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.131062 24119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:47.131173 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:47.131294 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:47.131330 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:47.131349 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622747131349 us; error 0 us; skew 500 ppm
I20250622 20:05:47.132098 22914 webserver.cc:469] Webserver started at http://127.22.96.131:42451/ using document root <none> and password file <none>
I20250622 20:05:47.132208 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:47.132261 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:47.132314 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:47.132634 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-2-root/instance:
uuid: "30bd4e0e1317422fa91b4930780e9991"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.132644 24113 heartbeater.cc:344] Connected to a master server at 127.22.96.190:40801
I20250622 20:05:47.132732 24113 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:47.132843 24113 heartbeater.cc:507] Master 127.22.96.190:40801 requested a full tablet report, sending...
I20250622 20:05:47.133059 23908 ts_manager.cc:194] Registered new tserver with Master: 7ca07ef4cd96472a8604113c07a20380 (127.22.96.130:38379)
I20250622 20:05:47.133360 23908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51976
I20250622 20:05:47.133687 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20250622 20:05:47.134120 24124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.134272 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.134313 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-2-root
uuid: "30bd4e0e1317422fa91b4930780e9991"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.134361 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:47.152999 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:47.153254 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:47.153441 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:47.153640 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:47.153667 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.153702 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:47.153730 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.157001 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.131:33871
I20250622 20:05:47.157058 24187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.131:33871 every 8 connection(s)
I20250622 20:05:47.160384 24188 heartbeater.cc:344] Connected to a master server at 127.22.96.190:40801
I20250622 20:05:47.160445 24188 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:47.160555 24188 heartbeater.cc:507] Master 127.22.96.190:40801 requested a full tablet report, sending...
I20250622 20:05:47.160791 23908 ts_manager.cc:194] Registered new tserver with Master: 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131:33871)
I20250622 20:05:47.161094 23908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51986
I20250622 20:05:47.161716 22914 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004429002s
I20250622 20:05:47.168363 22914 test_util.cc:276] Using random seed: -1627713942
I20250622 20:05:47.172439 23908 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51996:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 20:05:47.172566 23908 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:05:47.176892 24077 tablet_service.cc:1468] Processing CreateTablet for tablet 98b860b6d8314319900fecf60927fc47 (DEFAULT_TABLE table=test-workload [id=6bcc21af7c6444c59d3d0aa53e112e86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:47.177024 24077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98b860b6d8314319900fecf60927fc47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:47.177048 24152 tablet_service.cc:1468] Processing CreateTablet for tablet 98b860b6d8314319900fecf60927fc47 (DEFAULT_TABLE table=test-workload [id=6bcc21af7c6444c59d3d0aa53e112e86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:47.177170 24152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98b860b6d8314319900fecf60927fc47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:47.178109 24209 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380: Bootstrap starting.
I20250622 20:05:47.178292 24210 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991: Bootstrap starting.
I20250622 20:05:47.178694 24209 tablet_bootstrap.cc:654] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:47.178731 24210 tablet_bootstrap.cc:654] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:47.179245 24001 tablet_service.cc:1468] Processing CreateTablet for tablet 98b860b6d8314319900fecf60927fc47 (DEFAULT_TABLE table=test-workload [id=6bcc21af7c6444c59d3d0aa53e112e86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:47.179294 24209 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380: No bootstrap required, opened a new log
I20250622 20:05:47.179325 24210 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991: No bootstrap required, opened a new log
I20250622 20:05:47.179338 24209 ts_tablet_manager.cc:1397] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:47.179371 24210 ts_tablet_manager.cc:1397] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:47.179342 24001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98b860b6d8314319900fecf60927fc47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:47.179526 24210 raft_consensus.cc:357] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.179525 24209 raft_consensus.cc:357] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.179618 24210 raft_consensus.cc:383] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:47.179626 24209 raft_consensus.cc:383] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:47.179651 24210 raft_consensus.cc:738] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bd4e0e1317422fa91b4930780e9991, State: Initialized, Role: FOLLOWER
I20250622 20:05:47.179672 24209 raft_consensus.cc:738] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca07ef4cd96472a8604113c07a20380, State: Initialized, Role: FOLLOWER
I20250622 20:05:47.179709 24210 consensus_queue.cc:260] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.179711 24209 consensus_queue.cc:260] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.179878 24209 ts_tablet_manager.cc:1428] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250622 20:05:47.179911 24188 heartbeater.cc:499] Master 127.22.96.190:40801 was elected leader, sending a full tablet report...
I20250622 20:05:47.179966 24113 heartbeater.cc:499] Master 127.22.96.190:40801 was elected leader, sending a full tablet report...
I20250622 20:05:47.179879 24210 ts_tablet_manager.cc:1428] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250622 20:05:47.180797 24215 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Bootstrap starting.
I20250622 20:05:47.181190 24215 tablet_bootstrap.cc:654] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:47.181696 24215 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: No bootstrap required, opened a new log
I20250622 20:05:47.181735 24215 ts_tablet_manager.cc:1397] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:47.181854 24215 raft_consensus.cc:357] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.181907 24215 raft_consensus.cc:383] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:47.181921 24215 raft_consensus.cc:738] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d0e411419c4639ab8da0c41cc2b04f, State: Initialized, Role: FOLLOWER
I20250622 20:05:47.181972 24215 consensus_queue.cc:260] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.182116 24215 ts_tablet_manager.cc:1428] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.182152 24037 heartbeater.cc:499] Master 127.22.96.190:40801 was elected leader, sending a full tablet report...
I20250622 20:05:47.225921 24217 raft_consensus.cc:491] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:05:47.226018 24217 raft_consensus.cc:513] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.226215 24217 leader_election.cc:290] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ca07ef4cd96472a8604113c07a20380 (127.22.96.130:38379), 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131:33871)
I20250622 20:05:47.229429 24087 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98b860b6d8314319900fecf60927fc47" candidate_uuid: "06d0e411419c4639ab8da0c41cc2b04f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ca07ef4cd96472a8604113c07a20380" is_pre_election: true
I20250622 20:05:47.229429 24162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98b860b6d8314319900fecf60927fc47" candidate_uuid: "06d0e411419c4639ab8da0c41cc2b04f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd4e0e1317422fa91b4930780e9991" is_pre_election: true
I20250622 20:05:47.229555 24087 raft_consensus.cc:2466] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d0e411419c4639ab8da0c41cc2b04f in term 0.
I20250622 20:05:47.229557 24162 raft_consensus.cc:2466] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d0e411419c4639ab8da0c41cc2b04f in term 0.
I20250622 20:05:47.229753 23977 leader_election.cc:304] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d0e411419c4639ab8da0c41cc2b04f, 7ca07ef4cd96472a8604113c07a20380; no voters:
I20250622 20:05:47.229878 24217 raft_consensus.cc:2802] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:05:47.229941 24217 raft_consensus.cc:491] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:05:47.229969 24217 raft_consensus.cc:3058] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:47.230698 24217 raft_consensus.cc:513] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.230825 24217 leader_election.cc:290] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [CANDIDATE]: Term 1 election: Requested vote from peers 7ca07ef4cd96472a8604113c07a20380 (127.22.96.130:38379), 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131:33871)
I20250622 20:05:47.230970 24162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98b860b6d8314319900fecf60927fc47" candidate_uuid: "06d0e411419c4639ab8da0c41cc2b04f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bd4e0e1317422fa91b4930780e9991"
I20250622 20:05:47.231032 24162 raft_consensus.cc:3058] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:47.231559 24162 raft_consensus.cc:2466] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d0e411419c4639ab8da0c41cc2b04f in term 1.
I20250622 20:05:47.231649 24087 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98b860b6d8314319900fecf60927fc47" candidate_uuid: "06d0e411419c4639ab8da0c41cc2b04f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ca07ef4cd96472a8604113c07a20380"
I20250622 20:05:47.231704 24087 raft_consensus.cc:3058] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:47.231720 23976 leader_election.cc:304] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d0e411419c4639ab8da0c41cc2b04f, 30bd4e0e1317422fa91b4930780e9991; no voters:
I20250622 20:05:47.231839 24217 raft_consensus.cc:2802] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:47.231976 24217 raft_consensus.cc:695] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Becoming Leader. State: Replica: 06d0e411419c4639ab8da0c41cc2b04f, State: Running, Role: LEADER
I20250622 20:05:47.232029 24217 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.232347 24087 raft_consensus.cc:2466] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d0e411419c4639ab8da0c41cc2b04f in term 1.
I20250622 20:05:47.232563 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: term changed from 0 to 1, leader changed from <none> to 06d0e411419c4639ab8da0c41cc2b04f (127.22.96.129). New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:47.237125 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:05:47.237166 24087 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:05:47.237335 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:05:47.237408 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:05:47.256263 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.256687 24087 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250622 20:05:47.256690 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250622 20:05:47.256889 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250622 20:05:47.256975 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250622 20:05:47.257571 24219 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.257746 24087 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.258507 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.258649 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index -1 to 17, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.258731 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
W20250622 20:05:47.259346 23907 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 17: no extra replica candidate found for tablet 98b860b6d8314319900fecf60927fc47 (table test-workload [id=6bcc21af7c6444c59d3d0aa53e112e86]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250622 20:05:47.259482 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250622 20:05:47.259768 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250622 20:05:47.259815 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250622 20:05:47.260083 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250622 20:05:47.260234 24219 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.260347 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.260860 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.261240 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 17 to 18, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.261535 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.261979 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250622 20:05:47.262022 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250622 20:05:47.262157 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:47.262295 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250622 20:05:47.262822 24217 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.262962 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.262993 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.263787 23908 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 reported cstate change: config changed from index 18 to 19, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.263913 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.264238 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250622 20:05:47.264448 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250622 20:05:47.264595 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250622 20:05:47.264674 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250622 20:05:47.265156 24231 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.265265 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.265333 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.266162 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 19 to 20, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.266220 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.266534 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250622 20:05:47.266562 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250622 20:05:47.266703 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250622 20:05:47.266775 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250622 20:05:47.267395 24231 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.267561 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.267580 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.268272 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 20 to 21, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.268316 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.268924 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250622 20:05:47.268982 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250622 20:05:47.269135 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250622 20:05:47.269227 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250622 20:05:47.269719 24219 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.269863 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.269891 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.270699 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 21 to 22, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.270944 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.271299 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250622 20:05:47.271427 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250622 20:05:47.271569 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250622 20:05:47.271646 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250622 20:05:47.272157 24217 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.272432 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.272444 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.273133 23908 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 22 to 23, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.273290 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.273658 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250622 20:05:47.273928 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250622 20:05:47.273957 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250622 20:05:47.274133 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250622 20:05:47.274492 24219 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.274601 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.274749 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.275552 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 reported cstate change: config changed from index 23 to 24, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.275792 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.276099 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250622 20:05:47.276099 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250622 20:05:47.276330 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250622 20:05:47.276489 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250622 20:05:47.276857 24244 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.276983 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.277165 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.277757 23908 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 24 to 25, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.278138 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.278553 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250622 20:05:47.278589 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250622 20:05:47.278817 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250622 20:05:47.278932 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250622 20:05:47.279438 24231 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.279549 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.280385 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 reported cstate change: config changed from index 25 to 26, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.280426 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.280530 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.281157 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250622 20:05:47.281322 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250622 20:05:47.281374 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250622 20:05:47.281507 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250622 20:05:47.281934 24217 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.282084 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.282117 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.282886 23908 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 26 to 27, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.283257 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.283609 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 20:05:47.283764 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 20:05:47.283938 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 20:05:47.284224 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 20:05:47.284345 24217 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.284530 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.285184 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 27 to 28, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:47.285312 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.285208 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.285794 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 20:05:47.285955 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 20:05:47.286379 24087 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 20:05:47.286597 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 20:05:47.287189 24231 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.287295 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.288021 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 reported cstate change: config changed from index 28 to 29, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.288072 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.288203 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.288908 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 20:05:47.289170 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 20:05:47.289705 24244 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.289809 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 20:05:47.289829 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.289986 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 20:05:47.290256 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.290496 23894 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 21: aborting the task: latest config opid_index 29; task opid_index 21
I20250622 20:05:47.290746 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 reported cstate change: config changed from index 29 to 30, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.290978 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.291321 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250622 20:05:47.291711 24087 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250622 20:05:47.291605 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 20:05:47.291859 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 20:05:47.292296 24217 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.292431 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.293159 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 30 to 31, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:47.293202 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.293566 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250622 20:05:47.293574 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.294375 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250622 20:05:47.294385 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 20:05:47.294593 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 20:05:47.295029 24219 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.295141 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.295256 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.295938 23907 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 reported cstate change: config changed from index 31 to 32, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.296253 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.296661 24162 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250622 20:05:47.296802 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250622 20:05:47.296813 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 20:05:47.296952 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 20:05:47.297468 24219 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.297581 24086 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.298343 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 32 to 33, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.298404 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.298739 24162 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.298846 24086 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250622 20:05:47.299147 24217 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 20:05:47.299553 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20250622 20:05:47.299741 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 20:05:47.299724 24217 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.299853 24087 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.300132 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.300591 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 33 to 34, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: UNKNOWN } } }
I20250622 20:05:47.301358 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.301656 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250622 20:05:47.301664 24087 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20250622 20:05:47.301802 24231 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 20:05:47.301864 24219 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 20:05:47.302263 24244 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.302353 24087 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.302996 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.303134 23908 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 reported cstate change: config changed from index 34 to 35, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: NON_VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.303712 24011 consensus_queue.cc:237] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.304036 24087 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250622 20:05:47.304067 24161 raft_consensus.cc:1273] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 LEARNER]: Refusing update from remote peer 06d0e411419c4639ab8da0c41cc2b04f: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20250622 20:05:47.304186 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250622 20:05:47.304414 24244 consensus_queue.cc:1035] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250622 20:05:47.304682 24219 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.304854 24087 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.305017 24161 raft_consensus.cc:2953] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } } }
I20250622 20:05:47.305541 22914 timestamp_advancement-itest.cc:267] GCing logs...
I20250622 20:05:47.305613 22914 log_reader.cc:385] T 98b860b6d8314319900fecf60927fc47: removed 1 log segments from log reader
I20250622 20:05:47.305635 22914 log.cc:1079] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Deleting log segment in path: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root/wals/98b860b6d8314319900fecf60927fc47/wal-000000001 (ops 1-16)
I20250622 20:05:47.305778 23906 catalog_manager.cc:5582] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f reported cstate change: config changed from index 35 to 36, 30bd4e0e1317422fa91b4930780e9991 (127.22.96.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06d0e411419c4639ab8da0c41cc2b04f" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:47.306406 22914 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20250622 20:05:47.306456 22914 master.cc:561] Master@127.22.96.190:40801 shutting down...
I20250622 20:05:47.311457 23893 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 17: aborting the task: latest config opid_index 36; task opid_index 17
I20250622 20:05:47.317320 23894 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 33: aborting the task: latest config opid_index 36; task opid_index 33
I20250622 20:05:47.320717 23892 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 25: aborting the task: latest config opid_index 36; task opid_index 25
I20250622 20:05:47.322551 23894 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 23: aborting the task: latest config opid_index 36; task opid_index 23
I20250622 20:05:47.323736 23893 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 27: aborting the task: latest config opid_index 36; task opid_index 27
I20250622 20:05:47.329887 23893 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 19: aborting the task: latest config opid_index 36; task opid_index 19
I20250622 20:05:47.333705 23894 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b860b6d8314319900fecf60927fc47 with cas_config_opid_index 31: aborting the task: latest config opid_index 36; task opid_index 31
I20250622 20:05:47.524272 22914 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 1 LEADER]: Raft consensus shutting down.
I20250622 20:05:47.524389 22914 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:47.524464 22914 tablet_replica.cc:331] T 00000000000000000000000000000000 P e0ea4b1efa084ea1980c2dcb3c9184ee: stopping tablet replica
I20250622 20:05:47.537032 22914 master.cc:583] Master@127.22.96.190:40801 shutdown complete.
I20250622 20:05:47.538268 22914 tablet_server.cc:178] TabletServer@127.22.96.129:0 shutting down...
I20250622 20:05:47.616776 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:47.616968 22914 tablet_replica.cc:331] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: stopping tablet replica
I20250622 20:05:47.617028 22914 raft_consensus.cc:2241] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 LEADER]: Raft consensus shutting down.
I20250622 20:05:47.617116 22914 raft_consensus.cc:2270] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:47.629644 22914 tablet_server.cc:195] TabletServer@127.22.96.129:0 shutdown complete.
I20250622 20:05:47.630836 22914 tablet_server.cc:178] TabletServer@127.22.96.130:0 shutting down...
I20250622 20:05:47.880506 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:47.880682 22914 tablet_replica.cc:331] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380: stopping tablet replica
I20250622 20:05:47.880748 22914 raft_consensus.cc:2241] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:47.880831 22914 raft_consensus.cc:2270] T 98b860b6d8314319900fecf60927fc47 P 7ca07ef4cd96472a8604113c07a20380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:47.892807 22914 tablet_server.cc:195] TabletServer@127.22.96.130:0 shutdown complete.
I20250622 20:05:47.893772 22914 tablet_server.cc:178] TabletServer@127.22.96.131:0 shutting down...
I20250622 20:05:47.907994 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:47.908151 22914 tablet_replica.cc:331] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991: stopping tablet replica
I20250622 20:05:47.908231 22914 raft_consensus.cc:2241] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:47.908300 22914 raft_consensus.cc:2270] T 98b860b6d8314319900fecf60927fc47 P 30bd4e0e1317422fa91b4930780e9991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:47.919890 22914 tablet_server.cc:195] TabletServer@127.22.96.131:0 shutdown complete.
I20250622 20:05:47.920814 22914 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20250622 20:05:47.920928 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:47.922084 24246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.922150 24249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:47.922161 24247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:47.922379 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:47.922467 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:47.922500 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:47.922523 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622747922522 us; error 0 us; skew 500 ppm
I20250622 20:05:47.923192 22914 webserver.cc:469] Webserver started at http://127.22.96.129:39005/ using document root <none> and password file <none>
I20250622 20:05:47.923285 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:47.923326 22914 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 20:05:47.923978 22914 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.925230 24257 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.925354 22914 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250622 20:05:47.925405 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root
uuid: "06d0e411419c4639ab8da0c41cc2b04f"
format_stamp: "Formatted at 2025-06-22 20:05:47 on dist-test-slave-422w"
I20250622 20:05:47.925463 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1750622712029598-22914-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5294
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:47.961148 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:47.961398 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:47.961576 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:47.961858 24264 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250622 20:05:47.962546 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:05:47.962592 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250622 20:05:47.962628 22914 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:05:47.963063 22914 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:05:47.963096 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:47.963109 24264 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Bootstrap starting.
I20250622 20:05:47.965499 24264 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:05:47.965766 24264 tablet_bootstrap.cc:492] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Bootstrap complete.
I20250622 20:05:47.965875 24264 ts_tablet_manager.cc:1397] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250622 20:05:47.966003 24264 raft_consensus.cc:357] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.966085 24264 raft_consensus.cc:738] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d0e411419c4639ab8da0c41cc2b04f, State: Initialized, Role: FOLLOWER
I20250622 20:05:47.966121 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.129:45439
I20250622 20:05:47.966150 24264 consensus_queue.cc:260] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "06d0e411419c4639ab8da0c41cc2b04f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 45439 } } peers { permanent_uuid: "7ca07ef4cd96472a8604113c07a20380" member_type: VOTER last_known_addr { host: "127.22.96.130" port: 38379 } } peers { permanent_uuid: "30bd4e0e1317422fa91b4930780e9991" member_type: VOTER last_known_addr { host: "127.22.96.131" port: 33871 } }
I20250622 20:05:47.966514 24264 ts_tablet_manager.cc:1428] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250622 20:05:47.966759 24322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.129:45439 every 8 connection(s)
W20250622 20:05:47.969074 24323 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250622 20:05:47.973192 24277 tablet_service.cc:2912] Rejecting scan request for tablet 98b860b6d8314319900fecf60927fc47: Uninitialized: clean time has not yet been initialized
I20250622 20:05:47.973419 22914 timestamp_advancement-itest.cc:447] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20250622 20:05:47.973508 22914 tablet_server.cc:178] TabletServer@127.22.96.129:45439 shutting down...
I20250622 20:05:48.217568 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:48.217783 22914 tablet_replica.cc:331] stopping tablet replica
I20250622 20:05:48.217859 22914 raft_consensus.cc:2241] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 20:05:48.217924 22914 raft_consensus.cc:2270] T 98b860b6d8314319900fecf60927fc47 P 06d0e411419c4639ab8da0c41cc2b04f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:48.229712 22914 tablet_server.cc:195] TabletServer@127.22.96.129:45439 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (1220 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20250622 20:05:48.234230 22914 timestamp_advancement-itest.cc:460] Running with 1 replicas
I20250622 20:05:48.234390 22914 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.96.190:38039
I20250622 20:05:48.234544 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:48.235831 24330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.235889 24332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.235916 24329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.235983 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:48.236225 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:48.236260 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:48.236282 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622748236283 us; error 0 us; skew 500 ppm
I20250622 20:05:48.236930 22914 webserver.cc:469] Webserver started at http://127.22.96.190:41447/ using document root <none> and password file <none>
I20250622 20:05:48.237011 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:48.237058 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:48.237112 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:48.237365 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root/instance:
uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.238212 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:48.238648 24337 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.238777 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.238827 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root
uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.238865 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:48.252264 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:48.252461 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:48.255025 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.190:38039
I20250622 20:05:48.255628 24389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.190:38039 every 8 connection(s)
I20250622 20:05:48.255790 24390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:48.256803 24390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: Bootstrap starting.
I20250622 20:05:48.257087 24390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:48.257575 24390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: No bootstrap required, opened a new log
I20250622 20:05:48.257700 24390 raft_consensus.cc:357] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.257750 24390 raft_consensus.cc:383] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:48.257776 24390 raft_consensus.cc:738] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d5a7f2a20d4378ac9df187b0b4a5f3, State: Initialized, Role: FOLLOWER
I20250622 20:05:48.257881 24390 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.257936 24393 raft_consensus.cc:491] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:05:48.257997 24393 raft_consensus.cc:513] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.258071 24393 leader_election.cc:304] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d5a7f2a20d4378ac9df187b0b4a5f3; no voters:
I20250622 20:05:48.258088 24390 raft_consensus.cc:397] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:05:48.258117 24390 raft_consensus.cc:491] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:05:48.258152 24390 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:48.258185 24393 leader_election.cc:290] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20250622 20:05:48.258654 24390 raft_consensus.cc:513] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.258723 24390 leader_election.cc:304] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d5a7f2a20d4378ac9df187b0b4a5f3; no voters:
I20250622 20:05:48.258739 24393 raft_consensus.cc:2762] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250622 20:05:48.258759 24390 leader_election.cc:290] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:05:48.258814 24393 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:48.258888 24393 raft_consensus.cc:695] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 LEADER]: Becoming Leader. State: Replica: a9d5a7f2a20d4378ac9df187b0b4a5f3, State: Running, Role: LEADER
I20250622 20:05:48.258947 24390 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:05:48.258968 24393 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.259131 24393 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9d5a7f2a20d4378ac9df187b0b4a5f3. Latest consensus state: current_term: 1 leader_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER } }
I20250622 20:05:48.259162 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER } }
I20250622 20:05:48.259328 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:48.259263 24393 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:48.259749 24398 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:05:48.259874 24398 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:05:48.260365 22914 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 20:05:48.260421 24398 catalog_manager.cc:1349] Generated new cluster ID: c528739e257f44af9722afe5a98a71ba
I20250622 20:05:48.260450 24398 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:05:48.269659 24398 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:05:48.270109 24398 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:05:48.283113 24398 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: Generated new TSK 0
I20250622 20:05:48.283214 24398 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:05:48.292178 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:48.293444 24414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.293516 24411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.293558 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.293675 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:48.293752 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:48.293785 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:48.293805 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622748293805 us; error 0 us; skew 500 ppm
I20250622 20:05:48.294461 22914 webserver.cc:469] Webserver started at http://127.22.96.129:38231/ using document root <none> and password file <none>
I20250622 20:05:48.294548 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:48.294595 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:48.294651 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:48.294970 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root/instance:
uuid: "1a04813d2a234fdf9c4152d66e7b0b3f"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.295892 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:48.296376 24419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.296481 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.296530 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root
uuid: "1a04813d2a234fdf9c4152d66e7b0b3f"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.296584 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:48.305279 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:48.305500 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:48.305641 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:48.305838 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:48.305866 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.305900 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:48.305927 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.308768 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.129:40615
I20250622 20:05:48.308821 24482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.129:40615 every 8 connection(s)
I20250622 20:05:48.311965 24483 heartbeater.cc:344] Connected to a master server at 127.22.96.190:38039
I20250622 20:05:48.312023 24483 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:48.312126 24483 heartbeater.cc:507] Master 127.22.96.190:38039 requested a full tablet report, sending...
I20250622 20:05:48.312354 24354 ts_manager.cc:194] Registered new tserver with Master: 1a04813d2a234fdf9c4152d66e7b0b3f (127.22.96.129:40615)
I20250622 20:05:48.312914 24354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48300
I20250622 20:05:48.313306 22914 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004355099s
I20250622 20:05:48.319914 22914 test_util.cc:276] Using random seed: -1626562391
I20250622 20:05:48.324040 24354 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48328:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 20:05:48.327891 24447 tablet_service.cc:1468] Processing CreateTablet for tablet a9cb6fbc602a409babf55d28248a46bd (DEFAULT_TABLE table=test-workload [id=394bfbae8a1b4c64ae340f11565ad1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:05:48.328008 24447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cb6fbc602a409babf55d28248a46bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:05:48.329185 24501 tablet_bootstrap.cc:492] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Bootstrap starting.
I20250622 20:05:48.329504 24501 tablet_bootstrap.cc:654] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Neither blocks nor log segments found. Creating new log.
I20250622 20:05:48.329982 24501 tablet_bootstrap.cc:492] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: No bootstrap required, opened a new log
I20250622 20:05:48.330024 24501 ts_tablet_manager.cc:1397] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:48.330163 24501 raft_consensus.cc:357] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.330210 24501 raft_consensus.cc:383] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:05:48.330231 24501 raft_consensus.cc:738] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a04813d2a234fdf9c4152d66e7b0b3f, State: Initialized, Role: FOLLOWER
I20250622 20:05:48.330298 24501 consensus_queue.cc:260] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.330336 24501 raft_consensus.cc:397] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:05:48.330359 24501 raft_consensus.cc:491] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:05:48.330389 24501 raft_consensus.cc:3058] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:05:48.330987 24501 raft_consensus.cc:513] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.331060 24501 leader_election.cc:304] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a04813d2a234fdf9c4152d66e7b0b3f; no voters:
I20250622 20:05:48.331172 24501 leader_election.cc:290] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:05:48.331238 24503 raft_consensus.cc:2802] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:05:48.331302 24501 ts_tablet_manager.cc:1428] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:48.331327 24483 heartbeater.cc:499] Master 127.22.96.190:38039 was elected leader, sending a full tablet report...
I20250622 20:05:48.331331 24503 raft_consensus.cc:695] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 LEADER]: Becoming Leader. State: Replica: 1a04813d2a234fdf9c4152d66e7b0b3f, State: Running, Role: LEADER
I20250622 20:05:48.331400 24503 consensus_queue.cc:237] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.331840 24354 catalog_manager.cc:5582] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a04813d2a234fdf9c4152d66e7b0b3f (127.22.96.129). New cstate: current_term: 1 leader_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:48.333937 22914 tablet_server.cc:178] TabletServer@127.22.96.129:0 shutting down...
I20250622 20:05:48.336170 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:48.336305 22914 tablet_replica.cc:331] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: stopping tablet replica
I20250622 20:05:48.336364 22914 raft_consensus.cc:2241] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 LEADER]: Raft consensus shutting down.
I20250622 20:05:48.336416 22914 raft_consensus.cc:2270] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:48.347972 22914 tablet_server.cc:195] TabletServer@127.22.96.129:0 shutdown complete.
I20250622 20:05:48.348932 22914 master.cc:561] Master@127.22.96.190:38039 shutting down...
I20250622 20:05:48.351438 22914 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 LEADER]: Raft consensus shutting down.
I20250622 20:05:48.351508 22914 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:48.351581 22914 tablet_replica.cc:331] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: stopping tablet replica
I20250622 20:05:48.363617 22914 master.cc:583] Master@127.22.96.190:38039 shutdown complete.
I20250622 20:05:48.365399 22914 timestamp_advancement-itest.cc:460] Running with 3 replicas
I20250622 20:05:48.365592 22914 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.96.190:36899
I20250622 20:05:48.365739 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:48.366745 24508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.366870 22914 server_base.cc:1048] running on GCE node
W20250622 20:05:48.366814 24510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.366833 24507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.367120 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:48.367157 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:48.367174 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622748367175 us; error 0 us; skew 500 ppm
I20250622 20:05:48.367857 22914 webserver.cc:469] Webserver started at http://127.22.96.190:36001/ using document root <none> and password file <none>
I20250622 20:05:48.367946 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:48.367990 22914 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 20:05:48.368521 22914 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250622 20:05:48.368958 24515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.369086 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.369127 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root
uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.369186 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:48.373530 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:48.373708 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:48.376180 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.190:36899
I20250622 20:05:48.376945 24567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.190:36899 every 8 connection(s)
I20250622 20:05:48.377453 24568 sys_catalog.cc:263] Verifying existing consensus state
I20250622 20:05:48.377771 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: Bootstrap starting.
I20250622 20:05:48.379164 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:05:48.379374 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: Bootstrap complete.
I20250622 20:05:48.379544 24568 raft_consensus.cc:357] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.379598 24568 raft_consensus.cc:738] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d5a7f2a20d4378ac9df187b0b4a5f3, State: Initialized, Role: FOLLOWER
I20250622 20:05:48.379637 24568 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.379681 24568 raft_consensus.cc:397] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:05:48.379703 24568 raft_consensus.cc:491] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:05:48.379724 24568 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:48.380215 24568 raft_consensus.cc:513] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.380267 24568 leader_election.cc:304] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d5a7f2a20d4378ac9df187b0b4a5f3; no voters:
I20250622 20:05:48.380363 24568 leader_election.cc:290] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 20:05:48.380419 24571 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:05:48.380502 24568 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:05:48.380594 24571 raft_consensus.cc:695] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 2 LEADER]: Becoming Leader. State: Replica: a9d5a7f2a20d4378ac9df187b0b4a5f3, State: Running, Role: LEADER
I20250622 20:05:48.380652 24571 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER }
I20250622 20:05:48.380863 24572 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER } }
I20250622 20:05:48.380882 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9d5a7f2a20d4378ac9df187b0b4a5f3. Latest consensus state: current_term: 2 leader_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d5a7f2a20d4378ac9df187b0b4a5f3" member_type: VOTER } }
I20250622 20:05:48.380947 24572 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:48.380954 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:05:48.381165 24578 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:05:48.381438 24578 catalog_manager.cc:671] Loaded metadata for table test-workload [id=394bfbae8a1b4c64ae340f11565ad1e6]
I20250622 20:05:48.381587 24578 tablet_loader.cc:96] loaded metadata for tablet a9cb6fbc602a409babf55d28248a46bd (table test-workload [id=394bfbae8a1b4c64ae340f11565ad1e6])
I20250622 20:05:48.381649 24578 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:05:48.381744 24578 catalog_manager.cc:1261] Loaded cluster ID: c528739e257f44af9722afe5a98a71ba
I20250622 20:05:48.381785 24578 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:05:48.381884 22914 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 20:05:48.382529 24578 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:05:48.382648 24578 catalog_manager.cc:5966] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3: Loaded TSK: 0
I20250622 20:05:48.382767 24578 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:05:48.383128 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:48.384150 24588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.384193 24589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.384290 24591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.384336 22914 server_base.cc:1048] running on GCE node
I20250622 20:05:48.384433 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:48.384464 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:48.384490 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622748384490 us; error 0 us; skew 500 ppm
I20250622 20:05:48.385159 22914 webserver.cc:469] Webserver started at http://127.22.96.129:39093/ using document root <none> and password file <none>
I20250622 20:05:48.385250 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:48.385288 22914 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 20:05:48.385792 22914 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250622 20:05:48.386185 24596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.386312 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.386353 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root
uuid: "1a04813d2a234fdf9c4152d66e7b0b3f"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.386406 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:48.410586 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:48.410847 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:48.411047 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:48.411865 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:05:48.411900 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250622 20:05:48.412403 22914 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:05:48.412451 24603 tablet_bootstrap.cc:492] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Bootstrap starting.
I20250622 20:05:48.412442 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250622 20:05:48.413889 24603 tablet_bootstrap.cc:492] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:05:48.414148 24603 tablet_bootstrap.cc:492] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Bootstrap complete.
I20250622 20:05:48.414273 24603 ts_tablet_manager.cc:1397] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250622 20:05:48.414399 24603 raft_consensus.cc:357] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.414476 24603 raft_consensus.cc:738] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a04813d2a234fdf9c4152d66e7b0b3f, State: Initialized, Role: FOLLOWER
I20250622 20:05:48.414532 24603 consensus_queue.cc:260] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.414579 24603 raft_consensus.cc:397] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:05:48.414608 24603 raft_consensus.cc:491] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:05:48.414629 24603 raft_consensus.cc:3058] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:05:48.415401 24603 raft_consensus.cc:513] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.415498 24603 leader_election.cc:304] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a04813d2a234fdf9c4152d66e7b0b3f; no voters:
I20250622 20:05:48.415620 24603 leader_election.cc:290] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 20:05:48.415650 24657 raft_consensus.cc:2802] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:05:48.415746 24657 raft_consensus.cc:695] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 2 LEADER]: Becoming Leader. State: Replica: 1a04813d2a234fdf9c4152d66e7b0b3f, State: Running, Role: LEADER
I20250622 20:05:48.415777 24603 ts_tablet_manager.cc:1428] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250622 20:05:48.415839 24657 consensus_queue.cc:237] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } }
I20250622 20:05:48.416157 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.129:42197
I20250622 20:05:48.416188 24665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.129:42197 every 8 connection(s)
I20250622 20:05:48.416569 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:48.417763 24671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.417847 24672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.418030 22914 server_base.cc:1048] running on GCE node
W20250622 20:05:48.418097 24674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.418243 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:48.418296 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:48.418314 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622748418314 us; error 0 us; skew 500 ppm
I20250622 20:05:48.419092 22914 webserver.cc:469] Webserver started at http://127.22.96.130:34821/ using document root <none> and password file <none>
I20250622 20:05:48.419175 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:48.419225 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:48.419278 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:48.419577 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-1-root/instance:
uuid: "34cbb7be74e143539a49629acd41463a"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.420428 24667 heartbeater.cc:344] Connected to a master server at 127.22.96.190:36899
I20250622 20:05:48.420493 24667 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:48.420601 24667 heartbeater.cc:507] Master 127.22.96.190:36899 requested a full tablet report, sending...
I20250622 20:05:48.420735 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250622 20:05:48.420873 24532 ts_manager.cc:194] Registered new tserver with Master: 1a04813d2a234fdf9c4152d66e7b0b3f (127.22.96.129:42197)
I20250622 20:05:48.420926 24532 catalog_manager.cc:5582] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a04813d2a234fdf9c4152d66e7b0b3f" member_type: VOTER last_known_addr { host: "127.22.96.129" port: 40615 } health_report { overall_health: HEALTHY } } }
I20250622 20:05:48.421270 24679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.421408 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.421463 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-1-root
uuid: "34cbb7be74e143539a49629acd41463a"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.421505 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:48.422140 24532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57254
I20250622 20:05:48.422497 24667 heartbeater.cc:499] Master 127.22.96.190:36899 was elected leader, sending a full tablet report...
I20250622 20:05:48.445696 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:48.445960 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:48.446136 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:48.446336 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:48.446364 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.446403 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:48.446429 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.449725 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.130:32917
I20250622 20:05:48.449805 24743 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.130:32917 every 8 connection(s)
I20250622 20:05:48.450111 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:05:48.451577 24750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.451593 22914 server_base.cc:1048] running on GCE node
W20250622 20:05:48.452105 24747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:05:48.452158 24748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:05:48.452339 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:05:48.452378 22914 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:05:48.452399 22914 hybrid_clock.cc:648] HybridClock initialized: now 1750622748452398 us; error 0 us; skew 500 ppm
I20250622 20:05:48.453240 22914 webserver.cc:469] Webserver started at http://127.22.96.131:46649/ using document root <none> and password file <none>
I20250622 20:05:48.453344 22914 fs_manager.cc:362] Metadata directory not provided
I20250622 20:05:48.453398 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:05:48.453439 24744 heartbeater.cc:344] Connected to a master server at 127.22.96.190:36899
I20250622 20:05:48.453454 22914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:05:48.453505 24744 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:48.453697 24744 heartbeater.cc:507] Master 127.22.96.190:36899 requested a full tablet report, sending...
I20250622 20:05:48.453886 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-2-root/instance:
uuid: "13e674c267de4e0280b606524faa044c"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.453919 24532 ts_manager.cc:194] Registered new tserver with Master: 34cbb7be74e143539a49629acd41463a (127.22.96.130:32917)
I20250622 20:05:48.454257 24532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57270
I20250622 20:05:48.454838 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250622 20:05:48.455308 24755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.455426 22914 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.455472 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-2-root
uuid: "13e674c267de4e0280b606524faa044c"
format_stamp: "Formatted at 2025-06-22 20:05:48 on dist-test-slave-422w"
I20250622 20:05:48.455523 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfv7YOD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1750622712029598-22914-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:05:48.462520 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:05:48.462771 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:05:48.462981 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:05:48.463171 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:05:48.463199 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.463233 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:05:48.463259 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:05:48.466765 22914 rpc_server.cc:307] RPC server started. Bound to: 127.22.96.131:34089
I20250622 20:05:48.466817 24818 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.96.131:34089 every 8 connection(s)
I20250622 20:05:48.469596 24819 heartbeater.cc:344] Connected to a master server at 127.22.96.190:36899
I20250622 20:05:48.469651 24819 heartbeater.cc:461] Registering TS with master...
I20250622 20:05:48.469731 24819 heartbeater.cc:507] Master 127.22.96.190:36899 requested a full tablet report, sending...
I20250622 20:05:48.469959 24532 ts_manager.cc:194] Registered new tserver with Master: 13e674c267de4e0280b606524faa044c (127.22.96.131:34089)
I20250622 20:05:48.470252 24532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57284
I20250622 20:05:48.470257 22914 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003345509s
I20250622 20:05:48.476955 22914 test_util.cc:276] Using random seed: -1626405350
I20250622 20:05:48.481279 22914 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250622 20:05:48.481362 22914 tablet_server.cc:178] TabletServer@127.22.96.129:0 shutting down...
I20250622 20:05:48.483685 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:48.483908 22914 tablet_replica.cc:331] stopping tablet replica
I20250622 20:05:48.483968 22914 raft_consensus.cc:2241] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 2 LEADER]: Raft consensus shutting down.
I20250622 20:05:48.484022 22914 raft_consensus.cc:2270] T a9cb6fbc602a409babf55d28248a46bd P 1a04813d2a234fdf9c4152d66e7b0b3f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:48.495726 22914 tablet_server.cc:195] TabletServer@127.22.96.129:0 shutdown complete.
I20250622 20:05:48.496591 22914 tablet_server.cc:178] TabletServer@127.22.96.130:0 shutting down...
I20250622 20:05:48.498831 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:48.510386 22914 tablet_server.cc:195] TabletServer@127.22.96.130:0 shutdown complete.
I20250622 20:05:48.511230 22914 tablet_server.cc:178] TabletServer@127.22.96.131:0 shutting down...
I20250622 20:05:48.513878 22914 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 20:05:48.525070 22914 tablet_server.cc:195] TabletServer@127.22.96.131:0 shutdown complete.
I20250622 20:05:48.525949 22914 master.cc:561] Master@127.22.96.190:36899 shutting down...
I20250622 20:05:48.528174 22914 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 2 LEADER]: Raft consensus shutting down.
I20250622 20:05:48.528240 22914 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a9d5a7f2a20d4378ac9df187b0b4a5f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 20:05:48.528322 22914 tablet_replica.cc:331] stopping tablet replica
I20250622 20:05:48.540230 22914 master.cc:583] Master@127.22.96.190:36899 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (310 ms)
[----------] 4 tests from TimestampAdvancementITest (36510 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (36510 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20250622 20:05:48.544582 22914 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5260: suppressed but not reported on 7 messages since previous log ~1 seconds ago
I20250622 20:05:48.544621 22914 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20250622 20:05:48.544653 22914 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:536: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20250622 20:05:48.544682 22914 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 5 messages since previous log ~1 seconds ago
I20250622 20:05:48.544708 22914 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 13 messages since previous log ~1 seconds ago
I20250622 20:05:48.544763 22914 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:346: suppressed but not reported on 560 messages since previous log ~36 seconds ago